PaperlayerView exact revision on arXiv

Source-linked AI summary

Pedestrian Models for Autonomous Driving Part II: High-Level Models of Human Behavior

Fanta Camara, Nicola Bellotto, Serhan Cosar, Florian Weber, Dimitris Nathanael, Matthias Althoff, Jingyuan Wu, Johannes Ruenz, André Dietrich, Gustav Markkula, Anna Schieben, Fabio Tango, Natasha Merat, Charles W. Fox

arXiv:2003.11959v2cs.ROcs.GTcs.HCcs.LGcs.SYeess.SY

TL;DR

Autonomous vehicles need to anticipate pedestrians' destinations, paths, and responses to vehicle actions, but high-level psychological and social modeling remains an open research question. This review surveys behavior models across those levels and concludes that optimal walking models are well developed while translating qualitative behavior models into practical control algorithms remains unfinished.

  • Problem

    No unifying theory yet makes it easy to transfer pedestrian-model results across levels, while high-level psychological and social modeling still has conceptual issues to clarify.

    Source: §I · pp. 1–2

  • Method

    The review organizes pedestrian behavior research from individual trajectory and destination prediction through interaction models, including game-theoretic approaches.

    Source: §I · p. 1

  • Results

    Good control-theoretic models exist for optimal walking from known origin to known destination, but high-level social and psychological models remain an open question.

    Source: §I · p. 1, §VI · p. 15

  • Takeaways & Limitations

    The review identifies scope to integrate intelligent, goal-based, psychological, active, and interactive pedestrian models with simpler motion prediction.

    Source: §VI · p. 15

    Data-driven methods need verification for online real-time use, and computational cost increases significantly with the number of destination guesses.

    Source: F. Discussion · p. 7

Abstract

from arXiv · show

Autonomous vehicles (AVs) must share space with pedestrians, both in carriageway cases such as cars at pedestrian crossings and off-carriageway cases such as delivery vehicles navigating through crowds on pedestrianized high-streets. Unlike static obstacles, pedestrians are active agents with complex, interactive motions. Planning AV actions in the presence of pedestrians thus requires modelling of their probable future behaviour as well as detecting and tracking them. This narrative review article is Part II of a pair, together surveying the current technology stack involved in this process, organising recent research into a hierarchical taxonomy ranging from low-level image detection to high-level psychological models, from the perspective of an AV designer. This self-contained Part II covers the higher levels of this stack, consisting of models of pedestrian behaviour, from prediction of individual pedestrians' likely destinations and paths, to game-theoretic models of interactions between pedestrians and autonomous vehicles. This survey clearly shows that, although there are good models for optimal walking behaviour, high-level psychological and social modelling of pedestrian behaviour still remains an open research question that requires many conceptual issues to be clarified. Early work has been done on descriptive and qualitative models of behaviour, but much work is still needed to translate them into quantitative algorithms for practical AV control.

I. INTRODUCTION

Autonomous vehicles need pedestrian models spanning sensing, movement prediction, and social behavior, but connecting these levels remains difficult. Part II reviews higher-level behavior models and resources, while highlighting the immaturity of active interaction modeling and the absence of a unifying theory.

I. INTRODUCTION

  • AVs require models ranging from pedestrian detection and tracking to predictions of future movements and psychological or sociological behavior.Part II addresses higher-level behavior models, building on the sensing, detection, recognition, and tracking covered in Part I [33].Source: §I · p. 1
  • Active interaction models remain in their infancy, with rigorous study constrained by recently available data and the need for reliable pedestrian sensing and tracking.These models account for how the vehicle's future actions affect pedestrian responses and vice versa.Source: §I · pp. 1–2
  • No unifying theory yet makes it easy to transfer results across pedestrian-model levels, and machine-learning prediction performs less well than detection and recognition.The survey considers both empirical-science and machine-learning results; it proposes combining them as a direction for reducing over-fitting.Source: §I · p. 2
  • The review focuses on pedestrians moving under their own power on or near public highways, and notes that human-driving results do not directly transfer to pedestrian modeling.Its scope excludes people using cycles, wheelchairs, skates, or skateboards, and interactions among traffic participants without pedestrians.Source: §I · p. 2
  • The review covers individual movement and destination prediction, pedestrian-group interactions, game-theoretic situations, and datasets and simulators.Interaction models may not need game theory when pedestrians have different goals; competing goals, such as passing through a narrow passage, can make it useful.Source: §I · pp. 1, 3

II. BEHAVIOUR MODELS WITHOUT INTERACTION

This section reviews pedestrian movement prediction without interaction, progressing from unobstructed paths with known goals to models that use maps, obstacles, and contextual cues. Known-goal methods draw on several modeling approaches, but their setup excludes social effects.

II. BEHAVIOUR MODELS WITHOUT INTERACTION

  • The section moves from simple unobstructed-path models to uncertain-destination and event/activity models, all without interaction between agents.A previous review focused mainly on crossing intent and covered a more restricted range of trajectory-prediction models.Source: §II · p. 3

A. Unobstructed Walking Models with Known Goals

  • With known origins and destinations, inverse optimal control can reliably predict walking paths using a model that captures pedestrians' gradual turns and curved trajectories.The reviewed approaches model curved paths using travel-time, speed, and acceleration costs, or turning rates tied to visual angle and distance to the goal.Source: A. Unobstructed Walking Models with… · p. 3

B. Behaviour Prediction with Known Goals

  • Known-goal behavior prediction considers pedestrians moving around obstacles in a mapped static environment, without modeling social effects from other agents.Source: B. Behaviour Prediction with Known Goals · p. 3
  • Known-goal prediction spans particle-filter graphical models, context-based trajectory methods, 5s LSTMs, and models using goals, subgoals, obstacles, or turning maneuvers.The methods include GNSS hazard information, curbside and traffic-light context, path-planning similarities, and Monte Carlo trajectory generation.Source: B. Behaviour Prediction with Known Goals · pp. 3–4

C. Behaviour Prediction with Unknown Goals

Destination-unknown prediction combines goal inference with trajectory forecasting, while event/activity cues and pedestrian attributes offer additional behavioural evidence. The review describes demographic, attentional, social, cultural, and safety-related factors, but notes that many remain unintegrated into AV-ready models.

C. Behaviour Prediction with Unknown Goals

  • Because pedestrian destinations are rarely given, the described model reasons over possible destinations and infers them Bayesianly from prior observed trajectories, using environmental features when prior data are unavailable.Kitani et al. extended, with visual features and treated vision-system observations as noisy and uncertain.Source: C. Behaviour Prediction with Unknown… · p. 4

D. Event/Activity Models

  • Event/activity models use stereotyped behaviour sequences, head turns, and contextual or body-motion features to predict route choice and crossing behaviour.Real-world interaction studies identify head direction, pavement position, and hand gestures as predictive crossing features,.Source: D. Event/Activity Models · p. 5

E. Effects of Pedestrian Class on Trajectory

  • Age and gender studies report differences in crossing time, speed, risk, and facility preferences, though some differences are attributed to health and physical ability.Older pedestrians were reported to take longer and make more head movements; studies also report age- and gender-associated crossing differences,,,,.Source: E. Effects of Pedestrian Class on… · pp. 5–6
  • Distraction, group membership, and clothing may relate to crossing behaviour, including unsafe crossings, group assertiveness, and possible risk compensation.One study found a third of observed pedestrians distracted by phones, with distracted pedestrians more likely to cross unsafely and walk faster; high-visibility clothing may also relate to assertive behaviour,.Source: E. Effects of Pedestrian Class on… · p. 6
  • Cultural classification is proposed as potentially predictive when based on observable features, but the review notes that such classification is problematic and politicised.The review suggests considering observable-feature classifications if they improve road-interaction predictions.Source: E. Effects of Pedestrian Class on… · p. 6

F. Discussion

The discussion distinguishes established single-pedestrian path prediction from harder destination-unknown forecasting, and identifies computational and interpretability challenges. It also frames pedestrian interaction models as microscopic or macroscopic, while noting that psychological findings remain difficult to translate into AV algorithms.

F. Discussion

  • Known-origin-and-destination path models work for unobstructed walking and obstacles, but their assumptions limit real-world AV applicability; unknown destinations remain an open prediction problem.Uncertain-destination models can average known-destination models using destination predictions.Source: F. Discussion · p. 6
  • Long-term prediction methods include deep learning, but computational cost rises with destination guesses, horizon, and training data, while black-box decisions complicate incident analysis and liability.The review distinguishes short-term horizons of around 1–2 seconds from long-term horizons of around 5–6 seconds.Source: F. Discussion · p. 7
  • Demographic and visible class cues may inform behaviour, but the review says these findings are not yet translated into AV algorithms; emotion-state prediction remains conjectural.The review notes that emotion datasets are too small for machine learning and anticipates larger datasets from improving face and body-language recognition [33].Source: F. Discussion · p. 7

III. PEDESTRIAN INTERACTION MODELS

  • Figure 5 summarizes microscopic and macroscopic pedestrian models, which represent individuals versus aggregate entities such as crowds or flows.The review identifies interaction models as a distinct subject after single-pedestrian prediction.Source: Figure 5 · p. 7, §III · p. 7

A. Microscopic Models

Microscopic models describe individual pedestrian motion and interaction, using physical, cellular, or queuing approaches as well as proxemics. The review then surveys pairwise, crossing, and group interactions, including data-driven methods and their differing modeling assumptions.

A. Microscopic Models

  • Microscopic behaviour models include physical, cellular-based, and queuing-network families, generally balancing attraction toward goals against repulsion from people and obstacles.Proxemics adds personal-space utilities beyond collision avoidance, with binary collision-only and zonal formulations.Source: A. Microscopic Models · p. 7
  • Physical models represent utility or social forces, cellular models assign movement costs on grids, and queuing networks model pedestrians as flows through facilities.Social-force formulations include individual and group forces; cellular automata use cell occupancy and discrete movement rules, while queuing networks have been used in evacuation studies.Source: A. Microscopic Models · p. 8
  • Two-agent models capture mutually influencing behaviours and include graphical, Gaussian-process, and learned trajectory methods.Examples include POMDP/MDP approaches,, kriging-based social correlations, and social LSTM pooling of neighboring states.Source: A. Microscopic Models · p. 8
  • Road-crossing models extend pedestrian movement models with crossing decisions, including gap acceptance and perceptual evidence accumulation about gaps, yielding, signals, and eye contact.The perceptual-decision model qualitatively reproduced bimodal waiting times; a distance-ratio heuristic sufficed for most dataset scenarios, while remaining cases could require game theory.Source: A. Microscopic Models · p. 8
  • Group-interaction models treat groups as more than two pedestrians, smaller and more cohesive than crowds, and are designed for delivery robots navigating crowded pedestrianized areas between groups.Methods include local and global movement patterns, Gaussian-process crowd interaction, an LSTM long-term trajectory-prediction model, and GAN or deep-network trajectory-prediction methods,, [181].Source: A. Microscopic Models · p. 9
  • Other interaction approaches use desired direction and speed heuristics or social forces to reproduce crowd phenomena such as lane formation.Discrete-choice models also represent routes as sequences of links and have been used for shopping-street behaviour,.Source: A. Microscopic Models · p. 9

B. Macroscopic Models

Macroscopic models represent crowds as aggregate continuous flows, while smooth particle hydrodynamics bridges individual pedestrians and crowd density. The review contrasts these approaches with microscopic interaction models and notes trade-offs in precision, computational demand, and application domain.

B. Macroscopic Models

  • Macroscopic models replace individual pedestrians with an aggregate crowd behaving as a continuous fluid with average flow speed.Early fluid-dynamic models characterize pedestrian groups through average position, speed, and intended velocity.Source: B. Macroscopic Models · p. 9
  • Smooth particle hydrodynamics treats pedestrians individually but aggregates them into a density at each time, moving particles according to macroscopic velocity.The approach is described as a hybrid of microscopic and macroscopic models,.Source: B. Macroscopic Models · p. 10

C. Discussion

  • Macroscopic models suit crowd behaviour, especially evacuation, whereas microscopic models are generally preferred for pedestrian interaction; more detailed interaction models require more computation.Cellular models suit minimal movement choices when collision representation is unnecessary, and physical models are useful in settings with many interactions.Source: C. Discussion · p. 10, B. Macroscopic Models · p. 10

IV. GAME THEORETIC AND SIGNALLING MODELS

The section traces how game-theoretic models address increasingly complex pedestrian interactions, from reasoning about others’ plans to vehicle–pedestrian control. It also highlights unresolved challenges in selecting strategies and applying these models in practice.

A. Game Theory Interaction Models

  • First-order models describe one pedestrian’s behavior and second-order models let that pedestrian model another’s, while higher-order reasoning can regress infinitely.Game theory offers an analytic way to compute the infinite limit of these higher-order models.Source: A. Game Theory Interaction Models · p. 10
  • Game theory is less common in active AV control, and descriptive models may leave multiple Nash equilibria without selecting among them.A Nash equilibrium is a set of probabilistic strategies where no player would change strategy given the others’ strategies.Source: A. Game Theory Interaction Models · p. 10
  • A second-order truncation of a continuous game-theoretic model produced pedestrian flows resembling those observed at some Japanese crossings.Source: A. Game Theory Interaction Models · p. 10
  • Finite-set trajectory models and are descriptive rather than real-time controllers because they require post-hoc goal locations and do not model responses during interaction.Turnwald et al. instead let one player choose a trajectory first and the other respond to its initial motion.Source: A. Game Theory Interaction Models · pp. 10–11
  • Ma et al. [135] predict interacting trajectories from one image using fictitious play and learned utilities, but performance degrades when destination goals are unknown.Their utilities are learned from observed trajectories rather than specified from behavioral theory.Source: A. Game Theory Interaction Models · p. 11
  • The sequential chicken model represents AV–pedestrian choices at unsignalized intersections as decelerate-or-continue actions, with equilibrium selection by meta-strategy convergence.Laboratory fitting found θ to be the ratio between the utility of avoiding a collision and the utility of saving time,.Source: A. Game Theory Interaction Models · p. 11
  • Group and crowd models apply game theory to detecting conversational F-formations, simultaneous pedestrian–car interactions, and evacuation decisions, where cooperation brought pedestrians greater benefits.Source: A. Game Theory Interaction Models · p. 11

B. Signalling Interaction Models

Signalling models extend interaction beyond physical movement to information exchanged between pedestrians and vehicles. The reviewed work covers awareness and pedestrian cues, eHMI designs and effects, and unresolved challenges in communicating clearly.

B. Signalling Interaction Models

  • Signalling models extend physical interaction by letting pedestrians and AVs model each other’s acts of giving and receiving information.A mutual-awareness model distinguishes being unaware, factually aware, and actively attending; these levels correspond to bodily cues, implicit coordination, and explicit signals.Source: B. Signalling Interaction Models · p. 11
  • Signalling raises an open-ended question about higher-order knowing and showing, while early models have begun representing some levels of these beliefs.Eye contact is described as intuitively implying an infinite stack of mutual knowledge.Source: B. Signalling Interaction Models · pp. 11–12
  • Head orientation may signal pedestrian awareness, whereas eye contact may signal driver awareness to the pedestrian, helping explain differing findings about crossing behavior.Head orientation was sufficient for drivers to infer pedestrian intent in 52% of observed interactions; mutual eye contact occurred in 13%, and explicit signalling in 2%.Source: B. Signalling Interaction Models · pp. 11–12
  • Pedestrian signal settings and timing affect crossing decisions; sudden speed changes can surprise conflicting vehicles and create hazardous situations,,.After pedestrian flashing green, a model found a higher probability of stopping on longer crosswalks and a significant difference in pedestrian speeds.Source: B. Signalling Interaction Models · p. 12
  • The full game theory of vehicle-to-pedestrian signalling remains undeveloped, and the lack of two-way communication may reduce pedestrians’ confidence and perceived safety.The area includes patent activity around displays, but how best to use them to convey information is not understood.Source: B. Signalling Interaction Models · p. 12
  • Designers are advised to consider automation status, intended manoeuvres, environmental perception, and cooperation capabilities when deciding what an AV communicates.Communication may use eHMIs or vehicle movement, including advancing, slowing early, or stopping short; displays include projections, high-resolution screens, direct light, animation, icons, and text.Source: B. Signalling Interaction Models · p. 12
  • Studies report mixed effects of novel eHMIs on crossing, while other findings associate their presence with calmer, safer feelings and clearer vehicle-yield intent,,.Habibovic et al., argue that vehicles should communicate intentions rather than issue commands.Source: B. Signalling Interaction Models · p. 12
  • In ambiguous multi-pedestrian situations, undirected eHMI signals can be mistaken as addressed to someone else, supporting directed communication or no signal.Without an eHMI, pedestrians base crossing decisions on the approaching vehicle’s kinematics.Source: B. Signalling Interaction Models · p. 12

C. Discussions

The discussion identifies computational and translation challenges for advanced interaction models, while signalling research and eHMI evaluation remain immature. For validation, higher-level social-interaction data are scarce relative to visual and trajectory datasets.

C. Discussions

  • Game-theoretic models can be computationally expensive, and it remains unclear which theoretical solutions admit tractable algorithms.V2V transport studies and multi-robot systems have mature game-theory streams, but these have not generally been unified or applied to AV–pedestrian modelling.Source: C. Discussions · p. 13
  • Signalling remains a research frontier: explicit-signalling game-theoretic models are absent, and qualitative findings still need translation into AV algorithms.Whether eye contact is relevant or useful remains debated even in high-level psychology research.Source: C. Discussions · p. 13
  • Most proposed eHMI concepts lack detailed user studies, leaving their effects on pedestrian behavior, emotion, safety, experience, and acceptance unclear.Findings may differ with the eHMI concept, traffic scenario, and communication strategy.Source: C. Discussions · p. 13
  • Many conceptual eHMI solutions come from industry and use visual communication, a channel used in traffic and suited to longer distances in busy environments.Source: C. Discussions · p. 13

A. Pedestrian Datasets

  • High-quality annotated data are comparatively scarce for social interactions, although studies and collected physical and social events, communication cues, and interaction winners.Examples of visual resources include detection datasets such as CityPersons and re-identification datasets such as CUHK01, CUHK02, and CUHK03.Source: A. Pedestrian Datasets · p. 13
  • Trajectory resources cover vehicle-influenced pedestrian motion, interactive scenarios across multiple countries [227], and forecasting collections such as Argoverse Motion Forecasting.The cited Argoverse collection contains 324,557 scenarios, each five seconds long; the InD dataset provides trajectories for thousands of road users at four German intersections.Source: A. Pedestrian Datasets · pp. 13–14

B. Vehicle Datasets

The section reviews vehicle datasets and pedestrian–vehicle simulators, then concludes that individual walking models are more mature than models of social and interactive behavior.

B. Vehicle Datasets

  • BDDV is described as the largest publicly available vehicle dataset, with 10k hours of driving videos; KITTI has one hour of urban driving video, and nuScenes contains 1k scenes from two cities.nuScenes combines RGB, LIDAR, and RADAR data.Source: B. Vehicle Datasets · p. 14

C. Pedestrian and Driving Simulators

  • Pedestrian, vehicle, and combined pedestrian–vehicle simulators are reviewed, but closed hardware designs and source code impede replication and motivate open-source simulators.The review notes that the Godot game and VR engine may soon be usable for this purpose.Source: C. Pedestrian and Driving Simulators · p. 14
  • Pedestrian VR simulators study perception and decisions around virtual vehicles, while combined simulators model both pedestrian and vehicle behavior for design, policy, and AV testing.Examples include multi-agent VR setups and crosswalk or intersection simulations.Source: C. Pedestrian and Driving Simulators · pp. 14–15

VI. CONCLUSIONS

  • The review does not recommend software implementations for these active research areas; conceptual issues must be clarified before mathematical interfaces and standardized software can follow.Source: §VI · p. 15
  • The review describes good control-theoretic models for walking between known origins and destinations, with gradual turning producing smooth curves rather than straight paths.It also reports recent success inferring likely destinations from historical data and partial trajectories.Source: §VI · p. 15
  • Models become less well understood when other agents are included; game-theoretic approaches to higher-order beliefs have only begun to find optimal behavior under simplifying assumptions.Source: §VI · p. 15
  • The review notes a shift from psychology-informed empirical models toward big-data approaches that often predict behavior without theoretical explanations.Source: §VI · p. 15
  • Signalling between pedestrians and vehicles has been studied qualitatively, but algorithmic strategies for real-time AV control remain an open question.Examples include flashing headlights, horns, and custom communication lights.Source: §VI · p. 15

I. QUALITY OF CITATIONS

The section uses citation counts as an indication of the cited papers’ quality, presenting counts per paper and by publication year.

I. QUALITY OF CITATIONS

II. PEDESTRIAN TRAJECTORY AND INTERACTION MODELS

The section’s table summarizes pedestrian trajectory-prediction and interaction models, listing examples that use recurrent networks, planning methods, and decision processes.

II. PEDESTRIAN TRAJECTORY AND INTERACTION MODELS

  • The table covers both pedestrian trajectory prediction and interaction models.Source: Table I · p. 22
  • Listed prediction approaches include LSTM-based methods, including a mixture-density output layer and particle filtering, and event/activity models.Source: §II · p. 22
  • For interaction modeling, the entries include a time-state aggregated POMDP and an MDP with joint random-walk policy sampling.Source: §II · p. 22

III. DATASETS

The section inventories pedestrian and vehicle datasets, including resources for detection, tracking, trajectory prediction, and pedestrian–vehicle interactions.

III. DATASETS

  • The pedestrian-dataset inventory includes the Caltech Pedestrian Benchmark, the Multi-Person PoseTrack Dataset, and the MOTChallenge Benchmark.Source: §III · p. 24
  • Trajectory resources include ETHZ BIWI and UCY Zara, while the LIDAR trajectory dataset contains 20 interacting-person trajectories.Source: §III · p. 24
  • The inventory includes 204 annotated pedestrian–vehicle interactions at an unsignalized intersection and observer annotations for crossing or stopping.Source: §III · p. 24
  • Vehicle datasets include BDDV, with about one hour of daytime video from one city.Source: §III · p. 25
Loading 2003.11959v2…