Context-Aware Causal Gaze Forecasting for Human-Vehicle Interaction During In-Cabin Tracking Dropouts

I. INTRODUCTION

• Large head rotations can hide the driver’s eyes from dashboard trackers, and the tracker is unavailable for 8.5% of naturalistic driving time.These losses occur during mirror checks, shoulder checks, lane changes, and intersection scanning.Source: §I · pp. 1–2
• Causal gaze recovery forecasts during a dropout using only information available up to that instant, unlike offline imputation that uses observations from both sides of a gap.The online system cannot access measurements that have not yet occurred.Source: §I · p. 2
• The synchronized GazeSense and Neon recordings use Neon gaze only as supervision and evaluation targets, never as predictive input.The outward-facing Neon scene camera supplies the synchronized visual stream used as scene input.Source: §I · p. 2
• The released dataset contains genuine rotation-induced gaze dropouts from ten drivers, preserving real failure timing, duration, and head-motion patterns.This enables evaluation on real rather than synthetically generated tracker failures.Source: §I · p. 2
• CCGF combines pre-dropout gaze and head-motion history with visual context, and is compared with causal forecasting and non-causal imputation methods.Live scene updates provide an optimistic lower-error case; Frozen input provides a worst-case upper bound.Source: §I · p. 2

A. Driver Gaze Estimation from Observable Eyes

• Gaze-estimation methods improve robustness to pose and visibility variation, but still rely on some visual evidence of the eyes.The cited driver-gaze studies use eye, face, head-pose, or related geometric cues; their errors are not directly comparable across datasets and metrics.Source: A. Driver Gaze Estimation from… · p. 2

B. Missing Data and Causal Forecasting

• BRITS, SAITS, GP-VAE, and CSDI reconstruct gaps using observations after the missing interval, so they are offline references rather than online predictors.Causal forecasting baselines instead receive only pre-dropout GazeSense history and no future GazeSense observations or reference gaze.Source: B. Missing Data and Causal Forecasting · pp. 2–3
• The causal comparisons include DLinear, TimesNet, SegRNN, and a past-only CSDI protocol, alongside offline imputers.The supplied passage describes DLinear, TimesNet, and SegRNN as general-purpose forecasting architectures.Source: B. Missing Data and Causal Forecasting · pp. 2–3
• The studied setting combines no current eye-based observation during tracker loss with no access to post-recovery measurements.It is strictly causal forecasting during naturally occurring remote-tracker failure.Source: B. Missing Data and Causal Forecasting · p. 3

A. Dataset and Problem Setup

• The recordings comprise 10.5 hours from ten drivers, with 2,047 naturally occurring GazeSense head_lost events before benchmark filtering.Head_lost intervals account for 8.5% of recorded driving time.Source: A. Dataset and Problem Setup · p. 3
• Each event is paired with the preceding 60 frames, about two seconds, of causal gaze and head-motion context.The synchronized streams are expressed in a common 1600×1200 Neon scene coordinate system.Source: A. Dataset and Problem Setup · p. 3
• Eligible failures last 3–300 frames, require at least 70% valid Neon reference gaze, and require a valid GazeSense observation immediately before failure.The preceding context must also have at least 50% GazeSense availability and temporal continuity through the context–failure interval.Source: A. Dataset and Problem Setup · p. 3
• Scene-conditioned events require causal scene representations for at least 90% of failure frames, no older than 100 ms, yielding 2,047 eligible events.Frozen evaluation additionally requires a valid pre-dropout scene representation.Source: A. Dataset and Problem Setup · p. 3

B. Causal Forecasting Formulation

• The forecasting formulation uses the final valid GazeSense frame before loss, the failure interval, causal scene representations, and Neon reference gaze.The predictor receives no future GazeSense measurements, future scene observations, or Neon gaze targets; failure duration is unknown at inference.Source: B. Causal Forecasting Formulation · p. 3
• Live scene input updates during tracker loss, whereas Frozen input holds the final pre-dropout scene state fixed.A scene-free variant disables the visual pathway for an ablation.Source: B. Causal Forecasting Formulation · p. 3

C. CCGF Architecture

• The history pathway encodes 60 pre-dropout GazeSense frames with a unidirectional LSTM and conditions its final state on elapsed failure time.Missing GazeSense measurements are not recursively supplied after tracking is lost.Source: C. CCGF Architecture · p. 3
• The scene pathway projects DINOv3 features to 64 dimensions and processes them with a unidirectional LSTM, updating on Live observations or repeating the final pre-dropout representation when Frozen.Invalid scene observations do not advance the scene pathway.Source: C. CCGF Architecture · p. 4
• A learned scalar gate fuses history and scene states, with fallback to the history pathway when valid visual context is unavailable.The gate endpoints correspond to history-only and scene-only prediction.Source: C. CCGF Architecture · p. 4
• The fused representation yields a 24×32 spatial probability distribution whose center of mass is the predicted gaze.The distribution’s spatial spread provides an uncertainty estimate and contributes to the probabilistic training objective.Source: C. CCGF Architecture · p. 4

D. Training and Evaluation

• Synthetic failures augment training, while validation and test sets contain only naturally occurring tracker failures.Real events are sampled with probability 0.3; synthetic failures are generated from continuously tracked intervals and preserve the causal scene stream.Source: D. Training and Evaluation · p. 4
• The training objective combines probabilistic gaze, spatial heatmap, and valid pre-dropout context supervision.The heatmap and context losses receive weights 0.5 and 0.1, selected through nested driver-disjoint tuning on development drivers.Source: D. Training and Evaluation · p. 4
• Nested driver-disjoint LODO holds out one driver for testing and selects configurations within the remaining development drivers.The selected model is retrained on the full development set; recordings from each participant remain in one partition, and learned-model evaluation uses three seeds.Source: D. Training and Evaluation · p. 4
• Causal temporal baselines use the same 60-frame history, forecast in 30-frame blocks, and roll forward with their own predicted gaze if tracker loss continues.They receive no scene features or future observations; the true dropout duration is used only by the evaluator.Source: D. Training and Evaluation · p. 4
• Offline BRITS, SAITS, and GP-VAE references receive 60 post-recovery frames, so they can use future observations unavailable to online forecasting.They are treated as a separate, easier offline setting.Source: D. Training and Evaluation · p. 4
• Evaluation uses Euclidean pixel error and reports the mean of ten held-out drivers’ median framewise errors.Only naturally occurring failure frames with valid Neon reference gaze are included; angular error is complementary, and Neon gaze is not a model input.Source: D. Training and Evaluation · p. 5, Figure 2 · p. 5

IV. RESULTS

• Evaluation uses naturally occurring GazeSense head_lost events under the driver-disjoint LODO protocol.For each held-out driver, framewise prediction error is summarized by its median over eligible dropout frames.Source: §IV · p. 5
• The primary aggregation is the mean across ten test drivers of their median framewise prediction errors.Angular error is also reported.Source: §IV · p. 5
• Learned-model results are aggregated across three independent training seeds.Source: §IV · p. 5

A. Overall Causal Forecasting Performance

• CCGF-Live’s 175.7 px error is below DLinear (255.8 px), TimesNet (235.4 px), SegRNN (279.5 px), and CSDI (249.3 px).CCGF-Frozen reaches 210.8 px (12.91◦), below every causal baseline reported in Table I.Source: A. Overall Causal Forecasting… · p. 5
• Causal methods receive no future GazeSense observations or Neon gaze; temporal baselines use the 60-frame pre-dropout history, while CCGF also uses causal scene features.Table I compares complete forecasting systems; the scene-context contribution is isolated in the ablation.Source: A. Overall Causal Forecasting… · p. 5
• BRITS, SAITS, and GP-VAE are offline references that can use post-gap observations, yet their errors remain above CCGF under both scene conditions.For rotation-induced dropouts, frames after recovery resemble pre-dropout frames and carry little information about gaze during the excursion.Source: A. Overall Causal Forecasting… · pp. 5–6
• 175.7 px (10.52◦) is the CCGF-Live error, a 47.4% reduction from zero-order hold (334.2 px) and a 51.1% reduction from constant-velocity extrapolation (359.1 px).Table I reports learned-model results across ten LODO test folds and three independent training seeds.Source: A. Overall Causal Forecasting… · p. 5, Table I · p. 6

1) Scene Context and Reliability-Aware Fusion:

• 175.7 px with Live CCGF improves on the 263.9 px history-only LSTM; simple scene concatenation reaches 176.3 px.The 0.6 px difference indicates that much of the Live-condition gain comes from current visual context rather than gating alone.Source: 1) Scene Context and Reliability-Aware… · p. 6
• In Frozen conditions, CCGF reaches 210.8 px, improving by 33.5 px over simple concatenation (244.3 px).Stale scene input raises simple concatenation from 176.3 to 244.3 px and the ungated spatial-heatmap variant from 182.2 to 245.9 px.Source: 1) Scene Context and Reliability-Aware… · p. 6
• Removing within-dropout scene updates raises CCGF error from 175.7 px (10.52◦) to 210.8 px (12.91◦), a loss of 35.1 px and 2.39◦.Without scene information, CCGF reaches 258.4 px (15.93◦), close to the 263.9 px history-only LSTM.Source: 1) Scene Context and Reliability-Aware… · p. 6
• For held-out driver 7, the same 8,747 frames show median error rising from 153.4 px with Live updates to 189.6 px with Frozen input.P90 rises from 385.5 to 432.5 px; this is an illustrative single-driver comparison, while aggregate conclusions use the ten-driver evaluation.Source: Figure 3 · p. 6, 1) Scene Context and Reliability-Aware… · p. 6

2) Performance Across Dropout Duration:

• Live mean per-driver median errors are 165.6 px for short, 176.7 px for medium, 180.5 px for long, and 176.2 px for very-long dropouts.The very-long category contains only 61 events, so its estimate is interpreted cautiously.Source: 2) Performance Across Dropout Duration · p. 6
• Frozen errors are 189.4 px, 215.1 px, 211.2 px, and 199.8 px for short, medium, long, and very-long dropouts, respectively.Frozen is higher-error than Live in every duration group, with the largest differences for medium and long failures.Source: 2) Performance Across Dropout Duration · p. 6
• Live error rises only 14.9 px (approximately 9%) from short to long dropouts, while Frozen receives no new visual evidence after failure onset.Live continues to receive causally current scene observations as the last valid GazeSense measurement becomes older.Source: 2) Performance Across Dropout Duration · p. 6

C. Qualitative Analysis

• Fig. 4 compares CCGF-Live, CCGF-Frozen, DLinear, and TimesNet on matched frames spanning forward-facing and large-head-rotation conditions.Green circles mark Neon reference gaze, red circles model predictions, and CCGF heatmaps predicted spatial gaze distributions; these selected examples illustrate spatial behavior, not quantitative performance.Source: Figure 4 · p. 7, C. Qualitative Analysis · p. 7

D. Runtime and Online Feasibility

• CCGF batch-1 inference takes approximately 11.38 ms, including DINOv3 scene feature extraction, below the 33.3 ms period of a 30 Hz stream.Reported latency reflects the research implementation rather than a fully optimized streaming system; Neon gaze is not required online.Source: D. Runtime and Online Feasibility · p. 7

E. Limitations and Future Work

• Live uses a head-mounted Neon viewpoint that may encode head-motion cues, making it an optimistic egocentric condition rather than a direct fixed-camera deployment estimate.Frozen is a conservative condition because its scene input stops updating at the last forward-facing frame; the exact mapping to fixed vehicle cameras remains future work.Source: E. Limitations and Future Work · p. 7
• The framework uses GazeSense-derived driver state and visual context, while vehicle signals could add causal information during lane changes, turns, and intersection approaches.The listed signals include steering angle, vehicle yaw and pitch, acceleration, and turn-signal state.Source: E. Limitations and Future Work · p. 7
• The dataset contains ten drivers; larger, more diverse cohorts are needed to assess robustness across drivers, vehicles, road environments, and sensor configurations.Driver-disjoint evaluation and three training seeds provide a strict generalization test, but do not replace broader cohorts.Source: E. Limitations and Future Work · p. 7

• CCGF uses valid pre-dropout GazeSense gaze and head-pose history with causal scene observations, excluding future GazeSense measurements and Neon gaze from predictor inputs.Neon gaze serves only as an independent training and evaluation reference.Source: §V · p. 7
• 175.7 px (10.52°) with Live and 210.8 px (12.91°) with Frozen are CCGF’s mean per-driver median errors in driver-disjoint LODO evaluation.Source: §V · p. 7
• Under Live, CCGF reduces pixel error by 47.4% vs zero-order hold, 51.1% vs constant velocity, 31.3% vs DLinear, 25.4% vs TimesNet, 37.1% vs SegRNN, and 29.5% vs CSDI.Source: §V · pp. 7–8
• The Live scene stream follows the driver’s head, so its performance is an optimistic egocentric condition rather than a direct estimate of fixed-camera deployment accuracy.Source: §V · p. 8
• The central conclusion is that temporary remote-gaze loss does not necessarily mean all information about the driver’s attended region is lost; the authors identify fixed vehicle cameras, telemetry, and larger cohorts as next steps.Source: §V · p. 8