Source-linked AI summary
Predicting the Driver's Focus of Attention: the DR(eye)VE Project
Andrea Palazzi, Davide Abati, Simone Calderara, Francesco Solera, Rita Cucchiara
TL;DR
The paper addresses how to predict what drivers attend to in real-world scenes, where attention supports human driving and monitoring systems. It introduces the DR(eye)VE dataset and a multi-branch deep network combining raw video, motion, and semantics. The study finds shared gaze patterns across drivers and reports state-of-the-art prediction results, while noting that semantic input adds negligible improvement relative to color and motion in the tested ablation.
Problem
The paper asks how to estimate a driver’s focus of attention and identify critical scene regions for driving and driver-monitoring applications.
Method
The authors record and register eye-tracking data in DR(eye)VE and train a multi-branch network using visual appearance, optical flow, and semantic segmentation.
Results
The study finds gaze patterns shared across drivers and scenarios, and the proposed architecture achieves state-of-the-art results for predicting human attention in real-world driving sequences.
Takeaways & Limitations
Driver-attention prediction can reproduce common focus patterns and may support human-vehicle interaction and driver attention analysis.
Takeaways & Limitations
In the reported ablation, semantic input provides negligible improvement over color and motion, so it may be discarded under efficiency or real-time constraints.
Abstract
from arXiv · showhide
In this work we aim to predict the driver's focus of attention. The goal is to estimate what a person would pay attention to while driving, and which part of the scene around the vehicle is more critical for the task. To this end we propose a new computer vision model based on a multi-branch deep architecture that integrates three sources of information: raw video, motion and scene semantics. We also introduce DR(eye)VE, the largest dataset of driving scenes for which eye-tracking annotations are available. This dataset features more than 500,000 registered frames, matching ego-centric views (from glasses worn by drivers) and car-centric views (from roof-mounted camera), further enriched by other sensors measurements. Results highlight that several attention patterns are shared across drivers and can be reproduced to some extent. The indication of which elements in the scene are likely to capture the driver's attention may benefit several applications in the context of human-vehicle interaction and driver attention analysis.
1 INTRODUCTION
The paper studies drivers’ focus of attention using the DR(eye)VE dataset and a multi-branch model that combines visual appearance, motion, and scene semantics. Its analysis identifies shared gaze patterns and shows that attention varies with scene meaning, speed, landscape, and other driving conditions.
- Attention patterns: The study reports common gaze patterns across drivers and scenarios that a deep model can learn to predict in specific situations.These patterns motivate using data-driven attention analysis to model human focus during real-world driving.
- Dataset and motivation: The proposed network predicts car-centric salient regions through branches for raw visual information, optical flow, and semantic segmentation.The architecture is designed to integrate scene appearance, motion cues, and semantic information.
- Dataset and motivation: DR(eye)VE contains 555,000 frames from diverse driving sequences, with driver gaze registered to roof-mounted car views.The recordings span traffic conditions, landscapes, weather, lighting, and multiple drivers.
- Attention patterns: Dataset analysis finds a strong attraction of gaze toward the road’s vanishing point, while peripheral vision supports awareness of nearby objects.Drivers may disregard some signals, oncoming cars, and pedestrians when these lie outside the central focus.
- Attention patterns: Gaze shifts away from the central attractor as car speed and landscape change, and the dataset also reveals attraction toward specific semantic categories.Higher speed is associated with a smaller useful visual field in the reported analysis.
- Model design: The model uses intertwined coarse and refinement streams to reduce central-bias learning during training.One stream processes a severely cropped image, while the other predicts coarsely over the full image and then refines the result.
5 EXPERIMENTS
The experiments evaluate the multi-branch model against baselines, analyze its branches across driving conditions, and assess whether its predictions resemble human attention. Results favor the proposed architecture, while visual assessment suggests plausible fixation patterns.
- Evaluation setup: The evaluation uses Pearson’s Correlation Coefficient, Kullback–Leibler Divergence, and Information Gain, followed by branch ablation, attention-dynamics analysis, and human-perception assessment.Information Gain evaluates predicted maps against ground truth in the presence of a strong bias; the ablation study tests branch contributions and scenario-dependent interactions.
- Model comparison: The multi-branch architecture outperforms the compared methods on complete test sequences and acting subsequences.The comparison includes bottom-up approaches, MLNet, RMDN, and the previous RGB-only proposal.
- Driving conditions: Lower DKL indicates better predictions, with attention easier to predict on highways than downtown and the model more reliable in evening than morning or night scenarios.The analysis covers landscape, time of day, and weather conditions.
- Attention dynamics: As driving speed increases, predicted fixation maps shrink in spread, matching the trend observed in ground-truth maps.Spread is measured by fitting a two-dimensional Gaussian and reporting the determinant of its covariance matrix Σ.
- Branch ablation: The RGB branch makes the largest contribution, motion provides a beneficial slight improvement, and semantic segmentation adds only a negligible gain over image and motion.The semantic stream may therefore be discarded under efficiency or real-time constraints, although better segmentation could increase its benefit.
- Visual assessment: Participants achieved about 55% accuracy when guessing whether fixation maps came from humans or the model, close to random guessing.This result indicates that the generated attentional patterns appeared plausible to human observers.
6 CONCLUSIONS
The paper studies human attention dynamics in driving and proposes a multi-branch network to replicate drivers’ focus of attention from raw video. Its analysis identifies shared gaze patterns and links attention changes with driving conditions, while experiments report state-of-the-art results.
- Conclusions: The proposed multi-branch deep network captures factors identified in the prior attention analysis and predicts drivers’ focus of attention from raw video sequences.The model uses car-centric video as input and is described as the first model able to predict human attention in real-world driving sequences.
- Conclusions: The prior analysis identifies common gaze patterns across drivers and scenarios, alongside relations between attention changes and speed, lighting, weather, and landscape.These findings guide the network design and its training strategies.
- Conclusions: Experiments with the architecture and related training strategies yielded state-of-the-art results.The paper also notes that the car-centric video input could support integration with adopted ADAS technologies.
7 DR(EYE)VE DATASET DESIGN
The DR(eye)VE dataset comprises 74 five-minute sequences recorded across varied driving conditions, with experimental design intended to control potentially spurious correlations among key factors.
- Dataset design: The dataset contains 74 sequences of five minutes each, recorded under a variety of driving conditions.The sequence-level design is reported in the dataset table.
- Dataset design: Experimental design was used to rule out spurious correlations among driver, weather, traffic, daytime, and scenario.This design consideration was central to preparing the dataset.
8 VISUAL ASSESSMENT DETAILS
The visual assessment foveates videos from continuous attentional maps by approximating fixation locations and tracking preserved resolution. Model-generated maps produced more blurred videos than ground-truth maps, yet received higher perceived-safety ratings.
- Videoclip Foveation: Foveation uses SVIS to vary image resolution across the scene, preserving the highest resolution near fixation points and progressively blurring farther regions.SVIS builds a smooth resolution map from fixation points and selects image regions from different resolutions according to distance from the foveation point.
- Videoclip Foveation: Because the assessment uses continuous attentional maps, the 25 highest-valued map points are extracted as approximate fixation locations for SVIS.The choice corresponds to the temporal window of k = 25 frames used to build ground-truth fixation maps.
- Videoclip Foveation: Average post-foveation resolution is tracked across video frames, with higher vres indicating that more information is preserved.The measure averages the resolution map over the video’s frames.
- Videoclip Foveation: Model-predicted maps retained only 38% of the resolution of ground-truth-foveated videos, yet still produced higher perceived safety among assessment participants.Ground-truth attentional maps generated less blurred videos because their fixation locations were sparser.
9 PERCEIVED SAFETY ASSESSMENT
The assessment measures how safe observers feel when shown clips foveated according to different attentional maps. Model-predicted maps generally receive the highest perceived-safety ratings, especially for clips containing acting subsequences.
- Model-predicted attentional maps generally produce higher perceived-safety ratings than human or central-baseline maps.The central baseline remains highly competitive for non-acting videoclips, where it scores similarly to the model.
- In non-acting videoclips, the central-bias baseline scores similarly to the model prediction, and both automatic predictions outperform human groundtruth.
- For clips containing acting subsequences, the model prediction significantly outperforms both the center baseline and human groundtruth.
- Increasing false-positive rates at higher safety scores indicate that participants were sometimes tricked into believing safer clips came from human drivers.
10 DO SUBTASKS HELP IN FOA PREDICTION?
The model’s learned representations encode common driving subtasks. A linear SVM can distinguish four actions from its final-layer activations with high test accuracy.
- A linear SVM distinguishes turning left, turning right, going straight, and being still from the model’s final-layer activations.
- 90% test accuracy on 5,000 uniformly sampled videoclips shows that the network activations are highly discriminative for driving subtasks.
- The result supports the interpretation that the model leverages common driving subtasks to help produce its final focus-of-attention prediction.
11 SEGMENTATION
The segmentation branch is particularly useful when the driver’s gaze targets semantically meaningful road elements such as traffic lights and street signs.
- Only the semantic branch captures the driver’s gaze when attention is focused on traffic lights and street signs.
12 ABLATION STUDY
The ablation examples compare how the individual branches contribute to predicting the driver’s visual focus of attention. The RGB branch most strongly influences the overall network output.
- The RGB branch more heavily influences the overall network output than the other branches in the showcased examples.
13 ERROR ANALYSIS FOR NON-PLANAR HOMO-
The section analyzes how non-planar points violate homography assumptions and develops view-independent metric-error bounds alongside projection-error bounds.
- 13.1 The geometry of the problem: Homographies consistently map points on the inducing plane but misalign points on objects protruding from it.The cylinder base remains aligned across views, whereas its top produces a reprojection offset.
- 13.1 The geometry of the problem: Metric error e_W measures the world-space displacement caused by projecting a non-planar point onto the implicit plane.Unlike reprojection error, it is view-independent and unaffected by camera rotation relative to the plane.
- 13.3 Computing the error on a camera reference system: With an unknown inducing plane, metric error depends on the point x_2, height h, plane normal n, and camera separation v.The camera-reference formulation uses four independent quantities.
- 13.3 Computing the error on a camera reference system: When h ≠ 0, x_2 and v enter divided by h, so relative distances to the plane—not absolute positions—determine the error.The metric error also vanishes when either v or h tends to zero.
- 13.4 Working towards a bound.: Under the stated conditions, the world-space metric error is bounded by 2h.Observation 1 requires ||x_2|| ≥ 2||v||/|cos θ| and ||x_2|| > h.
- 13.4 Working towards a bound.: The corresponding image-plane reprojection error satisfies ||e_a|| ≤ 2f; with f = 350px, it remains below 200px.At 1080p resolution, this corresponds to precision up to 20% of the image.
14 THE EFFECT OF RANDOM CROPPING
The experiment evaluates the model’s reliance on random cropping by retraining with central crops and testing inputs shifted across a wide horizontal range.
- 14 THE EFFECT OF RANDOM CROPPING: The retrained multi-branch architecture uses central rather than random cropping for both input clips and ground-truth maps.At test time, each input clip is horizontally shifted within [−800, 800] pixels.
15 THE EFFECT OF PADDED CONVOLUTIONS IN LEARNING A CENTRAL BIAS
This section argues that padding supplies spatially localized statistics, allowing fully convolutional networks to learn absolute-position biases despite shared kernels.
- 15 THE EFFECT OF PADDED CONVOLUTIONS IN LEARNING A CENTRAL BIAS: Padding can reveal absolute position because padded values are constant and unrelated to the input feature map.The authors validate this mechanism with a toy task regressing a central white square on a black background.
16 ON FIG. 8
The histogram in Fig. 8 is constructed by collecting semantic classes inside fixation regions defined at multiple threshold levels.
- 16 ON FIG. 8: For each frame, pixel classes are collected from the segmentation map within the thresholded ground-truth fixation area.Increasing the threshold shrinks the area toward the actual fixation point.
17 FAILURE CASES
The paper reports failure cases in which the multi-branch architecture does not capture the ground-truth human attention. Related analyses examine receptive-field effects, padding-based central bias, and the construction of a class-occurrence count figure.
- 17 FAILURE CASES: The architecture fails to capture ground-truth human attention in several reported clips.These examples are presented as failure cases of the multi-branch architecture.
- 17 FAILURE CASES: Random cropping is evaluated as a way to prevent convolutional networks from learning a biased map from uniform inputs.The experiments compare output solutions and loss functions across receptive fields, using both uniform and noisy inputs.
- 17 FAILURE CASES: Receptive-field size affects whether padding statistics can break the uniformity of output feature maps.A central output pixel without access to padding statistics can activate differently from one whose receptive field exceeds the image borders.
- 17 FAILURE CASES: A separate figure represents the process used to count class occurrences for constructing Fig. 8.The passage directs readers to the text and paper for further details.