PaperlayerView exact revision on arXiv

Source-linked AI summary

Human Motion Trajectory Prediction: A Survey

Andrey Rudenko, Luigi Palmieri, Michael Herman, Kris M. Kitani, Dariu M. Gavrila, Kai O. Arras

arXiv:1905.06113v3cs.ROcs.CVcs.LG

TL;DR

Human motion trajectory prediction is important for autonomous systems operating around people, but existing methods face complex behavior and contextual influences. This paper surveys the literature, proposes a taxonomy separating modeling approaches from contextual cues, and reviews datasets, metrics, and benchmarking. It concludes that requirements depend strongly on application and use case, while current methods still have important contextual and evaluation limitations.

  • Problem

    Autonomous systems need to anticipate human motion for safe and effective operation, but human behavior reflects complex, partially unobservable influences and available benchmarking is difficult to compare.

  • Method

    The paper surveys motion prediction across domains and organizes methods by motion-modeling approach and contextual cues, while reviewing datasets, metrics, and benchmarking practices.

  • Results

    Requirements for motion prediction depend strongly on the application domain and use case, so absolute requirements cannot be concluded across scenarios.

  • Takeaways & Limitations

    Evaluation should use complex scenarios, geometric and probabilistic metrics, varied horizons and observation periods, realistic sensing conditions, and robustness experiments.

  • Takeaways & Limitations

    Many socially-aware methods assume people behave similarly and cooperatively, although real humans may optimize personal goals instead of joint strategies.

Abstract

from arXiv · show

With growing numbers of intelligent autonomous systems in human environments, the ability of such systems to perceive, understand and anticipate human behavior becomes increasingly important. Specifically, predicting future positions of dynamic agents and planning considering such predictions are key tasks for self-driving vehicles, service robots and advanced surveillance systems. This paper provides a survey of human motion trajectory prediction. We review, analyze and structure a large selection of work from different communities and propose a taxonomy that categorizes existing methods based on the motion modeling approach and level of contextual information used. We provide an overview of the existing datasets and performance metrics. We discuss limitations of the state of the art and outline directions for further research.

1 Introduction

Human motion trajectory prediction supports proactive perception, planning, control, and interaction across autonomous systems, but accurate forecasting is difficult because behavior reflects diverse, partially unobservable influences. This survey structures the literature by modeling approach and contextual cues, reviews benchmarks, and discusses open questions.

  • Motivation: Human motion prediction helps autonomous systems anticipate how scenes involving multiple agents will unfold for proactive perception, planning, control, and interaction.Applications include self-driving vehicles, service robots, and advanced surveillance systems.
  • Challenges: Accurate prediction is challenging because motion depends on intent, surrounding agents, social relations, norms, and environmental properties, many of which are not directly observable.These influences must be inferred from noisy perceptual cues or modeled from contextual information.
  • Scope and terminology: The survey focuses on ground-level 2D trajectory prediction for pedestrians, while also considering cyclists and vehicles; articulated motion, video frames, and activities are out of scope.A trajectory combines a sequence of positions with a timing law or velocity profile.
  • Survey contributions: The paper surveys methods across application domains, reviews datasets and metrics, analyzes benchmarking practices, and outlines open research challenges.It frames the discussion around whether evaluation is adequate, modeling choices still matter, and motion prediction is solved.
  • Survey organization: The taxonomy separates motion-modeling approaches from contextual cues, enabling methods with the same modeling category to be distinguished by their environmental and social awareness.Contextual cues are categorized by their relation to the target agent, other agents, and the static environment.

2 Taxonomy

The taxonomy separates motion prediction by modeling approach and contextual cues, treating these as distinct dimensions for organizing methods and their assumptions.

  • Taxonomy dimensions: The taxonomy classifies methods by motion modeling approach and contextual cues, with separate categories for how motion is represented and what information is used.The modeling dimension includes physics-based, pattern-based, and planning-based approaches.
  • Modeling approaches: Physics-based methods forward-simulate explicit dynamics, pattern-based methods learn trajectory patterns, and planning-based methods reason about goals and possible paths.These correspond to sense–predict, sense–learn–predict, and sense–reason–predict processes.
  • Literature trends: The survey reports a strong increase in related work during the last two years, particularly for pattern-based methods.Publication trends are color-coded by modeling approach.
  • Contextual cues: Contextual cues are grouped by their relation to the target agent, other agents, and static environmental properties.Target-agent cues include motion state, articulated pose, and semantic attributes.
  • Dynamic environment cues: Dynamic-environment categories distinguish unaware methods from methods accounting for individual agents or social grouping cues.The categories reflect increasing consideration of other agents in the scene.
  • Static environment cues: Static-environment categories range from unaware and obstacle-aware methods to map-aware and semantics-aware methods.These categories progressively incorporate obstacles, topometric maps, and environmental semantics or affordances.

3 Physics-based Approaches

Physics-based approaches predict motion by applying explicit dynamical models, ranging from simple kinematic models to multiple interacting motion modes and context-aware extensions.

  • Overview: Physics-based models describe motion using explicit dynamics and are commonly used within recursive Bayesian filters or multiple-model algorithms.They vary in motion type and in the complexity of their kinematic or dynamic formulations.
  • Model classes: Single-model approaches rely on one dynamical model, whereas multi-model approaches combine several dynamic modes with a mechanism for switching or fusion.The survey explicitly subdivides physics-based models into these two classes.
  • Single-model approaches: Kinematic models such as constant velocity, constant acceleration, coordinated turn, and bicycle models represent position, velocity, and acceleration without explicitly modeling forces.They are widely used because of their simplicity and acceptable performance under mild conditions and short horizons.
  • Single-model approaches: Dynamic models account for forces but become more challenging because force effects and detailed mechanisms are complex and not directly observable from sensory data.Examples include effects associated with wheels, gearboxes, engines, and friction.
  • Contextual extensions: Physics-based prediction recursively applies a dynamics model to the current state, while contextual information can be incorporated through additional forces, parameters, or state constraints.Most reviewed works make one-step-ahead predictions and ignore environmental context, with Zhu (1991) noted as an exception.
  • Map-based contextual cues: Map-based extensions encode road or free-space constraints through constrained estimates, pseudo-measurements, grid maps, or graph-based spatial representations.These representations support trajectory prediction in environments where agents follow roads or move through structured free space.
  • Dynamic environment cues: Map-based physics approaches discussed in this subsection consider only one target agent and neglect local interactions between multiple agents.The survey then introduces methods adding social situation awareness and joint prediction.
  • Dynamic environment cues: Social-force extensions combine goal attraction with repulsion from other agents and obstacles to model local interactions and more realistic behaviors.Later variants incorporate social relationships, collision avoidance, and group motion.

4 Pattern-based Approaches

Pattern-based approaches learn motion behavior from data using function approximators, with sequential models conditioning predictions on state histories. The survey covers local, location-independent, higher-order, continuous, and recurrent representations, including models of social and environmental context.

  • Pattern-based approaches learn motion dynamics from data using function approximators such as neural networks, hidden Markov models, and Gaussian processes.
  • Sequential Models: Sequential methods model each state as conditionally dependent on a sufficient statistic of previous states, often using Markov assumptions.
  • Sequential Models: Local transition models learn movement probabilities in grid cells or higher-level workspace representations such as sub-goals, graphs, and Voronoi diagrams.
  • Sequential Models: Gaussian-process models represent continuous, often multimodal transition dynamics, including change-point detection and online learning of previously unseen motion patterns.
  • Sequential Models: Location-independent methods predict motion in general free space, while higher-order sequential models learn longer temporal dependencies between states.
  • Sequential Models: Recurrent models, especially LSTMs, incorporate temporal history and can pool information about neighboring pedestrians, static surroundings, road infrastructure, or vehicle maneuvers.

5 Planning-based Approaches

Planning-based approaches predict motion by reasoning about sequential decisions and modeling humans as rational agents. They either apply predefined costs or infer costs and policies from demonstrations, with extensions for interactions and multimodal behavior.

  • Planning-based approaches follow a Sense-Reason-Act paradigm and model human motion as sequential decision-making by a rational agent.
  • Forward planning uses predefined reward or cost functions, whereas inverse planning infers cost or policy functions from observed behavior.
  • Forward Planning: Forward methods use hand-crafted path costs, goals, cost-to-go estimates, and probabilistic transition evaluation to generate goal-directed predictions.
  • Forward Planning: Planning methods incorporate interactions through local policy modifications, global planning with collision avoidance, or joint state-space planning for multiple agents.
  • Inverse Planning: Inverse planning learns reward functions or policies from observations using inverse optimal control, imitation learning, or neural models.
  • Inverse Planning: Under rigid-body-invariant feature, parameter, and output representations, inverse reinforcement learning can learn location-independent transferable motion models.
  • Inverse Planning: A differentiable imitation-learning model balances prediction precision, by avoiding obstacle areas, with diversity, by retaining multiple modes.

6 Contextual Cues

The survey organizes contextual cues around the target agent, other dynamic agents, and the static environment. It distinguishes interaction awareness and increasingly rich obstacle and map representations for predicting socially and geometrically constrained motion.

  • Contextual cues are categorized by whether they concern the target agent, other dynamic agents, or the static environment.
  • Cues of the Target Agent: Most prediction algorithms use the target agent’s current position and velocity as essential inputs.
  • Cues of Other Dynamic Agents: Interaction-aware prediction distinguishes unaware, individual-aware, and group-aware predictors according to how they model other agents.
  • Cues of Other Dynamic Agents: High-level social context, including heterogeneous interactions, social comfort, flow preference, and avoidance of dense areas, produces more precise predictions in varied cases.
  • Cues of the Static Environment: Environmental cues range from obstacle-free assumptions and implicit collision-free patterns to obstacle-aware, map-aware, and semantic-map representations.
  • Cues of the Static Environment: Obstacle-aware methods may fail in very cluttered environments because representing the scene as individual obstacles becomes complex.

7 Motion Prediction Evaluation

The survey reviews geometric and probabilistic metrics alongside benchmark datasets for evaluating trajectory prediction. It emphasizes application-appropriate measures, because geometric metrics cannot represent uncertainty or multimodality and different metrics can produce different results.

  • Evaluation design must compare prediction methods against application requirements using suitable metrics and datasets.
  • Geometric Accuracy Metrics: Geometric metrics include MED/ADE, PCA-Euclid, MEDP, DTW, MHD, THAU, LCS, CLEAR-MOTA, and FDE, each capturing different trajectory similarities.
  • Geometric Accuracy Metrics: FDE measures final-position error, but it is inappropriate when multiple future positions are possible.
  • Geometric Accuracy Metrics: Prediction Accuracy classifies trajectories as correct when predicted positions satisfy a specified criterion, such as a distance threshold.
  • Geometric Accuracy Metrics: Different metrics can produce substantially different results, so evaluation should not select measures to favor a particular method.
  • Probabilistic Accuracy Metrics: Geometric metrics cannot measure uncertainty or multimodal predictions, motivating probabilistic metrics that assess agreement with the distribution of human movements.
  • Probabilistic Accuracy Metrics: Minimum-displacement and related distributional metrics encourage coverage of multiple ground-truth modes, while top n% evaluation produces more stable results than top 1% evaluation.
  • Datasets: The survey selects widely reused datasets for benchmarking and adds recent datasets covering cyclists, vehicles, dedicated trajectory prediction, and accurate motion capture.

8 Discussion

The survey finds substantial progress and methodological diversity, but its three central questions remain unresolved, especially regarding evaluation consistency and whether prediction is solved.

  • The survey reports progress in prediction techniques across method diversity, performance, and application relevance.
  • Q1 is not confirmed because benchmarking lacks a systematic approach with common evaluation practices.
  • The discussion asks whether motion prediction is solved by revisiting application requirements and identifying open challenges for future research.

8.1 Benchmarking

The survey argues that current benchmarking does not support systematic comparison: datasets, metrics, scenarios, and robustness practices vary substantially. It recommends standardized, richer evaluations that condition results on prediction and scene characteristics.

  • Evaluation should combine appropriate scenarios, accuracy metrics, and robustness testing across interaction levels and maneuvering complexity.Relevant scenario factors include the number of interacting agents and the amount of maneuvering in the data.
  • Existing datasets span indoor and outdoor settings but often lack sufficiently long trajectories, balanced interactions, and semantic information about static and dynamic entities.
  • Probabilistic metrics such as NLL or KLD-derived log-loss are appropriate for multimodal, uncertain forecasts, while geometric metrics remain common for short-term prediction.
  • Metric formulations and names vary, and evaluating at a single arbitrary horizon obstructs comparison of relative precision across methods.
  • Very few authors address robustness to sensing errors, tracking deficiencies, self-localization uncertainties, or map changes.
  • Q1 is not confirmed: despite numerous metrics, datasets, and experiment designs, benchmarking lacks systematic common evaluation practices.
  • Recommended evaluations use complex scenarios, complete geometric and probabilistic metrics, and precision conditioned on horizons, observation periods, and scene complexity.
  • A standard benchmark with shared scenarios and metrics is critical for fair comparison, with TrajNet identified as an initial attempt using ETH, UCY, and Stanford Drone Dataset trajectories.

8.2 Modeling Approaches

The survey finds no universally preferable modeling approach: physics-, pattern-, and planning-based methods trade off interpretability, data needs, generalization, runtime, and prediction horizon. Q2 therefore remains unconfirmed.

  • Modeling approaches differ in inherent strengths and limitations, particularly in how they incorporate contextual cues.
  • Physics-based methods suit explicitly modeled dynamics and context, but their usability is limited to short horizons and relatively obstacle-free environments.
  • Pattern-based methods handle complex unknown dynamics and comparatively large horizons, but require ample location-specific data and may generalize poorly across changed maps.
  • Planning-based methods work well when agent goals and an environmental map are explicit, producing better long-term predictions than physics-based techniques in such cases.
  • Q2 is not confirmed because studies insufficiently compare approaches, which also differ in complexity and efficiency when incorporating contextual information.
  • Planning-based methods may fail to incorporate dynamic cues through reward functions, requiring modifications such as Jump Markov Processes, local trajectory adaptations, or game-theoretic methods.

8.3 Application Domains

Application requirements vary substantially across service robotics, self-driving vehicles, and surveillance, especially in prediction horizon, context, and evaluation. Reported performance is therefore difficult to compare directly across datasets and metrics.

  • Service robotics: Service robots favor fast, short-term predictions for local planning and collision avoidance, whereas global planning requires long-term, multi-hypothesis forecasts.Short-term horizons are described as 1–2 seconds; global planning may require 15–20 seconds and richer contextual reasoning.
  • Service robotics: 0.19–0.4 m ADE is reported on ETH and UCY for a 4.8 s horizon, compared with 0.53 m ADE for linear velocity projection.These results concern simpler pedestrian datasets.
  • Self-driving vehicles: Self-driving research emphasizes maneuver recognition and trajectory prediction for pedestrians, cyclists, and vehicles, with multimodality, uncertainty, and coordination increasingly important.Scenarios include pedestrian crossings, cyclist turns, and multiple possible road directions.
  • Self-driving vehicles: Context-based SLDS prediction improved lateral position errors by 0.9 m versus context-free SLDS and 1.1 m versus basic LDS for pedestrian stopping up to 1 s ahead.A live demonstration reported evasive vehicle action up to 1 second earlier with the context-based model.
  • Surveillance: Surveillance uses trajectory prediction for tracking, future-position distributions, customer-flow analysis, and anomaly detection, but results remain difficult to compare across heterogeneous evaluations.Reported crowded-scene results range from 0.08–1.2 m ADE and 0.081–2.44 m FDE over approximately 0.8–4.8 seconds.
  • Cross-domain assessment: Use-case requirements differ enough that absolute requirements cannot be concluded; basic emergency braking for intelligent vehicles is presented as an industrialized, solved use case.Further conclusions require examining performance, robustness, generalization, and benchmarking across domains.

8.4 Future Directions

Future work should improve contextual understanding, generalization, robustness, and integration of motion prediction with planning and control. The survey also identifies automatic goal inference and multi-predictor systems as promising directions.

  • Enhanced contextual cues: Modern methods increasingly use machine learning to model context-dependent patterns, complex environments, and future motion directly from visual input.The trend extends beyond Kalman filtering with simple motion models.
  • Enhanced contextual cues: Semantic understanding of static environments remains relatively unexplored, while socially aware methods still often assume homogeneous, cooperative behavior.The survey highlights a need to represent high-level social attributes and non-cooperative personal goals.
  • Goal and intent inference: Long-term prediction should incorporate intentions and surrounding context, because context-agnostic patterns are mainly useful for short horizons.Many planning-based methods also depend on predefined goals.
  • Goal and intent inference: Automatic goal inference could identify possible goals online and predict how agents reach them, addressing settings where goals are unknown or numerous.This direction is proposed as an alternative to relying on a fixed goal set.
  • Robustness and generalization: Combining multiple predictors may improve robustness in undefined or changing situations where different methods offer complementary strengths.The survey cites a multiple-predictor framework as a conceptually interesting example.
  • Robustness and generalization: Transfer learning, invariant motion rules, and domain adaptation are suggested to improve generalization to new environments.Domain adaptation is proposed when data from several environments are available.
  • Robustness and integration: Integrating prediction into planning and control is identified as necessary for improving overall system robustness.Many papers motivate prediction by safety and efficiency but leave its use in robot action planning outside their scope.

9 Conclusions

The survey organizes human motion trajectory prediction across application domains through a taxonomy based on motion models and contextual cues. Despite more than two decades of research, prediction remains difficult and the paper calls for continued work on the identified challenges.

  • Conclusions: The paper surveys multiple application domains and structures motion prediction techniques by motion models and input contextual cues.It also reviews trajectory-prediction tasks and discusses the state of the art and future research directions.
  • Conclusions: Despite two decades of research and more than 200 listed prediction methods, forecasting future motion remains difficult.The conclusion frames the survey as a way to increase visibility and stimulate further research.
Loading 1905.06113v3…