Source-linked AI summary
Joint Attention in Autonomous Driving (JAAD)
Iuliia Kotseruba, Amir Rasouli, John K. Tsotsos
TL;DR
Autonomous driving requires safe interaction with traffic participants, yet communication and perception remain challenging. The paper proposes JAAD as a dataset for studying joint attention and traffic behavior, with videos spanning varied pedestrian situations but fewer interactions with other drivers.
Problem
Urban autonomous driving must meet high safety and precision standards, while communication lapses can contribute to erroneous behavior.
Source: §I · p. 1Method
The paper proposes a dataset for studying joint attention and records participant, scene-condition, and behavior annotations for each video clip.
Source: §I · p. 2, §IV · p. 5Results
JAAD videos cover varied pedestrian situations, mostly in urban areas, while clips involving other drivers are fewer.
Source: §IV · pp. 4–5Takeaways & Limitations
The dataset’s two annotation types make it suitable for pedestrian and car detection and research on joint attention and non-verbal communication.
Source: §V · p. 5
Abstract
from arXiv · showhide
In this paper we present a novel dataset for a critical aspect of autonomous driving, the joint attention that must occur between drivers and of pedestrians, cyclists or other drivers. This dataset is produced with the intention of demonstrating the behavioral variability of traffic participants. We also show how visual complexity of the behaviors and scene understanding is affected by various factors such as different weather conditions, geographical locations, traffic and demographics of the people involved. The ground truth data conveys information regarding the location of participants (bounding boxes), the physical conditions (e.g. lighting and speed) and the behavior of the parties involved.
I. INTRODUCTION
Urban autonomy still requires reliable interaction with traffic participants, but communication can fail and current vehicle- and pedestrian-facing approaches have practical constraints. The paper frames vehicle-to-vehicle and vehicle-to-pedestrian communication as joint attention and proposes a dataset for studying traffic behavior and scene complexity.
I. INTRODUCTION
- Urban autonomous driving remains unresolved because vehicles must meet high precision and safety standards while interacting with their environment.The cited account also identifies infrastructure and behavior regulation as challenges.Source: §I · p. 1
- In Google’s 2015 report, more than one-third of 341 disengagements were attributed to perception discrepancy, and about 10% to incorrect prediction or response to reckless behavior.The trials covered more than 424,000 miles; about 90% of disengagements occurred on busy streets.Source: §I · p. 1
- V2V and V2I communication relies on costly cellular technology, is less reliable than radar and cameras, and depends on all parties functioning properly.A malfunction in any involved communication device can lead to catastrophic safety issues.Source: §I · p. 1
- Pedestrian communication technologies include phone-based position and activity broadcasts, but technological and regulatory obstacles make near-term use seem infeasible.The phone approach could report whether a pedestrian is listening to music, texting, or on a call.Source: §I · p. 1
- A hood-mounted notification screen can signal a car’s intended action, but such explicit interfaces have been criticized as distracting and ineffective with multiple pedestrians.Other proposed interface options include sound or physical devices.Source: §I · pp. 1–2
- Prior studies connect pedestrian awareness and non-verbal cues such as eye contact or smiling with driver actions, while wireless communication has also been explored for traffic coordination.The cited work includes V2V and V2I communication for tasks such as cooperative cruise control and positioning.Source: §I · pp. 1–2
- The paper reframes vehicle-to-vehicle and vehicle-to-pedestrian communication as joint attention and proposes a dataset to study traffic-scene and behavioral complexity.It also discusses why existing approaches may be inadequate in this context.Source: §I · p. 2
II. AUTONOMOUS DRIVING AND JOINT ATTENTION
The paper applies joint attention to autonomous driving, where drivers and pedestrians interpret one another’s attention and intentions amid varied visual and social cues. It argues that scene context and visual robustness matter beyond recognizing objects, and notes limitations of deep learning for these demands.
II. AUTONOMOUS DRIVING AND JOINT ATTENTION
- Pedestrian-driver exchanges at crossings illustrate joint attention: pedestrian actions and a driver’s slowing can be observed and interpreted as yielding.The examples include pedestrians looking up, slowing, and then crossing after the car slows.Source: §II · p. 2, Figure 1 · p. 3
- Joint attention is established research in robotics and human-robot interaction, but the paper says it had not been explicitly discussed in autonomous driving.Much prior work focused on gaze, pointing, reaching, turn-taking, and social referencing, usually with stationary robots or heads.Source: §II · p. 2
- Traffic gestures require contextual interpretation because a wave may come from a policeman, a crossing pedestrian, or someone hailing a taxi.The paper also identifies lighting, weather, and scene clutter as visual processing challenges.Source: §II · p. 2
- Deep learning’s strong object-recognition performance does not establish causal relationships between observations and their context, limiting its fit for complex driving situations.The paper raises tasks such as merging with traffic, responding to jaywalking, and yielding as beyond simple object detection.Source: §II · p. 4
- Deep learning is also described as lacking robustness to visual-input changes, including minor pixel-level changes that can cause misclassification.Source: §II · p. 4
III. EXISTING DATASETS
Public autonomous-driving datasets largely target detection, tracking, mapping, and related tasks, with limited behavioral annotation; access to naturalistic driving data is also restricted. Existing datasets vary in their labels, from pedestrian boxes to scene and camera parameters.
III. EXISTING DATASETS
- Public driving datasets mainly serve mapping, navigation, and detection, while only a limited number support behavioral studies and naturalistic driving raw data is restricted to qualified researchers.The public may receive only depersonalized statistics from large naturalistic driving studies.Source: §III · p. 4
- KITTI includes videos of pedestrians, vehicles, and cyclists, but annotates no behaviors; Caltech provides pedestrian bounding boxes with temporal correspondence and occlusion labels.KITTI is used for 3D reconstruction, detection, tracking, and visual odometry; Caltech contains approximately 10 hours of urban driving.Source: §III · p. 4
- Berkeley provides pedestrian boxes for detection and tracking, while SSFM annotates object boxes and name tags for detection in street-view videos.Berkeley videos were collected from a stationary car at street intersections.Source: §III · p. 4
- The German Traffic Sign Detection Benchmark labels traffic-sign positions, EISATS supplies camera intrinsic parameters, and Daimler pedestrian datasets omit behavioral analysis.EISATS videos support stereo-vision and motion-analysis evaluation; Daimler datasets cover pedestrian detection tasks including segmentation, classification, and path prediction.Source: §III · p. 4
IV. THE JAAD DATASET
JAAD comprises urban-driving clips selected to study traffic-participant behavior and joint attention. It combines bounding-box and event-based textual annotations across varied scenarios, participants, and conditions, though other-driver interactions and non-daytime recordings are relatively sparse.
IV. THE JAAD DATASET
- JAAD contains 346 high-resolution clips of 5–15 seconds, extracted from approximately 240 hours of driving video collected in several locations.Two vehicles used wide-angle cameras mounted inside the cars below the rear-view mirror.Source: §IV · p. 4
- The clips cover varied urban traffic scenarios, including pedestrians looking back for a traffic gap, peeking around obstacles, and waiting to cross.Most footage is urban; only a few clips were filmed in rural locations.Source: §IV · p. 4
- Pedestrians appear across ages and in solo or group settings, including with children or pets; age and group context may affect walking pace and caution.The authors note elderly people and parents with children may walk more slowly and cautiously.Source: §IV · p. 5
- Interactions with other drivers are represented by fewer clips, mostly at uncontrolled intersections, parking lots, or during multi-lane turns.Source: §IV · p. 5
- Most videos were recorded during daytime, with only a few at night, sunrise, or sunset; the dataset also includes varied weather conditions.Sun glare at sunrise and sunset, and obstructed faces or limited road visibility in heavy snow or rain, complicate attention assessment.Source: §IV · p. 5
- Bounding boxes cover cars and pedestrians who interact with or require the driver's attention, and record frame number, coordinates, dimensions, and occlusion.Source: §IV · p. 5
- BORIS textual annotations record clip-level weather, time, location, pedestrian age and gender, and crosswalk designation, alongside labeled subjects and driver or pedestrian behaviors.The labels distinguish the in-car “Driver” from “Driver_other”; behavior labels include actions such as walking, standing, looking, and moving.Source: §IV · p. 5
V. CONCLUSION
The paper presents JAAD as a dataset for studying joint attention in autonomous driving, with two annotation types that also support pedestrian and car detection and research on nonverbal communication. Its event annotations distinguish duration-based states from brief point events.
V. CONCLUSION
- JAAD is presented as a joint-attention dataset whose two annotation types also suit pedestrian and car detection and research on social robotics and human non-verbal communication.Source: §V · p. 5
- Behavior annotations distinguish state events of arbitrary duration from point events lasting 0.1 seconds to mark quick glances or pedestrian gestures.Source: Table II · p. 6, §V · p. 6
- The BORIS example organizes a video identifier, clip-level variables, and timestamped events with subject, behavior, and status fields for recovering event sequences.Source: Figure 3 · p. 6, §V · p. 6