Source-linked AI summary
Agreeing to Cross: How Drivers and Pedestrians Communicate
Amir Rasouli, Iuliia Kotseruba, John K. Tsotsos
TL;DR
The paper addresses limited evidence about how pedestrians and drivers communicate during crossings and presents JAAD, a visual dataset paired with an analysis of joint attention and crossing behavior. More than 90% of pedestrians used attention before crossing, while crossing also depended on TTC, driver reaction, and crosswalk structure.
Problem
Urban autonomous vehicles must communicate with road users, but existing approaches do not fully account for contextual factors such as driver reaction, vehicle speed, or crossway structure.
Method
The authors introduce JAAD and analyze annotated pedestrian actions and visual attention, including looking and glancing, across varied crossing scenarios.
Results
More than 90% of cases involved pedestrian attention before crossing, while crossing depended on street structure, driver reaction, and TTC.
Takeaways & Limitations
Pedestrian crossing communication includes attention, explicit gestures, and responses to driver actions, rather than a single communication event determining crossing.
Abstract
from arXiv · showhide
The contribution of this paper is twofold. The first is a novel dataset for studying behaviors of traffic participants while crossing. Our dataset contains more than 650 samples of pedestrian behaviors in various street configurations and weather conditions. These examples were selected from approx. 240 hours of driving in the city, suburban and urban roads. The second contribution is an analysis of our data from the point of view of joint attention. We identify what types of non-verbal communication cues road users use at the point of crossing, their responses, and under what circumstances the crossing event takes place. It was found that in more than 90% of the cases pedestrians gaze at the approaching cars prior to crossing in non-signalized crosswalks. The crossing action, however, depends on additional factors such as time to collision (TTC), explicit driver's reaction or structure of the crosswalk.
I. INTRODUCTION
Fully autonomous urban driving remains unsolved partly because vehicles must communicate with road users amid formal rules, informal cues, and challenging conditions. This paper introduces JAAD and analyzes pedestrian crossing behavior and non-verbal communication.
- Urban autonomous driving remains unsolved, with communication among road users a major dilemma in chaotic traffic scenes.
- Pedestrians often use informal non-verbal communication and anticipation alongside official traffic rules when deciding whether to cross.
- JAAD is a novel visual dataset for detecting and analyzing pedestrian behavior while crossing or attempting to cross under varied conditions.
- The paper analyzes pedestrians’ actions and non-verbal cues across crossing scenarios, including effects associated with crosswalk structure, driver behavior, and vehicle distance.
A. Studies of driver and pedestrian interaction
Prior studies examine driver yielding, awareness, pedestrian decisions, and factors such as TTC, road geometry, weather, and eye contact. Existing predictive approaches often emphasize motion or crude body-language cues without incorporating crossing context.
- Psychological studies examine driver yielding, driver awareness, and pedestrian decision making before crossing events.
- Reported behavioral factors include vehicle speed, time to collision, vehicle gaps, road geometry, weather, crossing conditions, age, gender, and eye contact.
- Traffic interactions have often been treated mechanistically, with TTC combining vehicle speed and pedestrian distance to relate to crossing behavior.
- Drivers are more likely to yield when looked at by a pedestrian waiting to cross.
- Passive computer-vision approaches predict crossing actions from trajectories, velocities, group behavior, or body language.
- Body-language approaches associate head orientation with awareness but do not account for driver reaction, vehicle speed, or crossway structure.
B. Existing Datasets
Existing pedestrian datasets provide detection-oriented annotations, but the authors identify no dataset facilitating study of pedestrian crossing behavior. They therefore conduct a small-scale naturalistic driving study to extract non-verbal communication data across situations.
- KITTI, the Caltech benchmark, and the Daimler dataset provide pedestrian-detection ground truth such as bounding boxes, stereo information, sensor readings, and occlusion tags.
- The authors report no existing datasets designed to facilitate study of pedestrians’ crossing behavior.
- Relevant psychological data is often collected at selected locations through researchers’ direct observation.
- Because raw Naturalistic Driving Study data is restricted, the authors conducted a small-scale naturalistic driving study and extracted non-verbal communication data.
III. THE JAAD DATASET
JAAD comprises video clips from extensive driving footage covering varied urban scenarios, times, weather, lighting, and pedestrian interactions. The dataset combines targeted bounding-box annotations with textual behavioral annotations and metadata.
- 346 high-resolution JAAD video clips of 5–15 seconds were extracted from approximately 240 hours of driving videos collected across several locations.
- The clips primarily represent urban and suburban scenarios, with fewer rural examples and fewer interactions involving other drivers.
- Scenarios include individual and group crossings, pedestrian occlusion, and pedestrians walking along the road.
- Videos span different times of day and varied weather and lighting conditions, including challenging sun glare, snow, and rain.
- Annotations include bounding boxes for interacting or driver-relevant cars and pedestrians plus textual observations created with BORIS.
- Each clip records weather, time of day, pedestrian age and gender, location, and whether the crossing is designated.
IV. THE DATA
The dataset reveals highly varied pedestrian crossing behavior and organizes it into nine groups based on initial state and whether attention or crossing occurs. It also distinguishes two forms of visual attention by duration and purpose.
- More than 100 distinct action patterns occur in pedestrian crossing and no-crossing scenarios.Common sequences such as “standing, looking, crossing” and “crossing, looking” cover only half of completed crossings.
- The patterns are split into 9 groups according to the pedestrian’s initial state and whether attention or crossing is occurring.
- Attention is defined as the pedestrian’s first environmental assessment while expressing an intention to approaching vehicles.The paper treats this initial assessment as non-verbal communication.
- Looking typically lasts at least 1 second and may involve assessing the car or establishing eye contact, whereas glancing is a brief traffic assessment.Glancing usually lasts less than a second and occurs when the pedestrian is confident the vehicle poses little immediate danger.
V. OBSERVATIONS AND ANALYSIS
The analysis excludes incomplete or ambiguous crossing cases, while Table II presents the behavioral patterns observed in the data.
- Incomplete crossing scenarios are omitted because they do not show the full event and make behavior at the crossing point difficult to assess.
- Action cases are omitted because pedestrian intentions are ambiguous when people are far from or not approaching the curb.
- Table II lists the behavioral patterns observed in the data.
A. Forms of non-verbal communication
Pedestrians primarily communicate crossing intentions through looking or glancing, while other signals are rarer and often respond to driver actions. Their responses may appear as behavioral changes rather than explicit signals.
- In more than 90% of dataset crossing events, pedestrians use non-verbal communication, primarily looking or glancing toward approaching traffic.Looking accounts for 90% and glancing for 10% of the reported signals.
- Looking is the prominent signal at 90%, while glancing accounts for 10% of observed communication toward coming traffic.
- Nodding and hand gestures are rarer signals, usually performed in response to the driver’s action.Nodding conveys gratitude or acknowledgement; hand gestures convey gratitude or yielding.
- Pedestrian responses are often behavioral changes rather than explicit communication, such as slowing down or stopping.These changes can indicate noticing an approaching vehicle or that the driver is not yielding.
- Table III summarizes communication and responses while distinguishing primary attention from secondary attention during crossing.
B. Attention occurrence prior to crossing
Pedestrian attention before crossing varies with TTC, crosswalk designation, street width, and pedestrian age. Attention-free crossings are uncommon and concentrated in safer-looking contexts, while gaze duration changes nonlinearly with TTC.
- TTC measures how long an approaching vehicle takes to reach the pedestrian’s position if it maintains its speed and trajectory.
- About 10% of crossings occur without attention, with more than 50% of these cases occurring when TTC exceeds 10s.
- No attention-free crossings occur when TTC is below 2s, and attention-free crossing in the samples occurs only at non-designated crosswalks when TTC exceeds 6s.
- Full crossing events occur on streets ranging from 1 to 4 lanes, while below 3s TTC no attention-free crossings occur on streets wider than 2 lanes.
- Gaze duration increases with TTC up to a maximum safe threshold of 7s for adults and 8s for elderly pedestrians, then declines sharply.
- Elderly pedestrians spend on average about 1s longer looking before crossing than adults and children.
C. Crossing action post attention occurrence
Whether pedestrians cross after signaling attention depends on crosswalk structure and the driver’s response. Pedestrians are more likely to proceed when designated pathways or signals are present and when unsignalized drivers slow or stop.
- The study distinguishes non-designated, zebra, and traffic-signal crosswalks according to markings, signs, or signals that require drivers to stop.
- Pedestrians are less likely to cross after communicating their intention at non-designated crosswalks and more likely when a signal or dedicated pathway is present.
- Driver responses are grouped into maintaining or increasing speed, slowing down, and stopping.
- Without a traffic signal, pedestrians cross in most cases when drivers acknowledge their intention by slowing down or stopping.
- Some pedestrians cross while vehicles accelerate when TTC is very high, averaging 25.7s, or when congestion suggests the vehicle will soon stop.
VI. CONCLUSIONS
Pedestrians communicate crossing intentions primarily through attention toward approaching vehicles, while nodding and hand gestures are less common responses to driver actions. Crossing decisions also depend on street structure, driver reaction, and time to collision, motivating richer analysis of pedestrian gait and driver behavior.
- In more than 90% of cases, pedestrians use attention toward approaching vehicles to communicate an intention to cross.Looking is the most prominent form of attention-based communication.
- Nodding and hand gestures occur less often and commonly respond to the driver’s action by expressing gratitude, acknowledgement, or yielding.These explicit communication forms were observed in 15% of cases.
- Crossing depends on street structure, the driver’s reaction to communication, and time to collision rather than necessarily following the first communicated intention.
- Future analysis should examine pedestrian gait with and without attention and record driver gestures, eye movements, and vehicle-state changes.