Source-linked AI summary
CrowdTraj: A Benchmark for Dense Crowd Trajectory Prediction in Realistic Crowded Environments
Antonius Bima Murti Wijaya, Paul Henderson, Marwa Mahmoud
TL;DR
Existing trajectory benchmarks offer limited evaluation of dense, occluded CCTV crowds and often assume perfect trajectory inputs. CrowdTraj provides five such scenes for end-to-end evaluation from detection through tracking to forecasting, with pixel and real-world coordinates. Its evaluations report tracking accuracy challenges under heavy occlusion and greater computational effort for trajectory prediction.
Problem
Existing benchmarks largely assume perfect trajectories or feature sparse, less-occluded scenes, leaving dense CCTV crowd prediction insufficiently evaluated.
Source: Introduction · p. 1Method
CrowdTraj is a five-scene CCTV benchmark for end-to-end evaluation from detection through tracking to forecasting, with real-world coordinates for distance-based analysis.
Source: Introduction · p. 1Results
Tracking methods struggle with accuracy under heavy occlusion, while trajectory-prediction methods struggle more with computational effort; MART and modified SocialVAE show different strengths across settings.
Source: Conclusion and Future Research · p. 7Takeaways & Limitations
CrowdTraj exposes robustness to tracking noise and computational scalability as challenges that sparse-crowd benchmarks leave largely hidden.
Source: Introduction · p. 2The benchmark comprises five CCTV scenes, bounding its reported setting to those captured environments.
Source: Conclusion and Future Research · p. 7
Abstract
from arXiv · showhide
In real-world applications, pedestrian trajectory prediction models rely on inputs from detection and tracking systems. Prior trajectory prediction benchmarks either contain relatively sparse pedestrian interactions, assume perfect tracking inputs, or rely on overhead viewpoints that minimize occlusion and perspective distortion, limiting evaluation in realistic dense-crowd scenarios. We present CrowdTraj, a benchmark for pedestrian trajectory prediction in natural dense crowd scenes. Unlike previous datasets, CrowdTraj supports end-to-end evaluation from detection through tracking to trajectory prediction under severe occlusion in CCTV views. It also captures diverse, natural pedestrian behaviours, including abrupt directional changes rarely observed in existing benchmarks. CrowdTraj includes five diverse scenes, with an average of 1,146 unique pedestrians per scene, maximum frame-level densities ranging from 114 to 372 pedestrians, and over 3.2 million annotated head bounding boxes. CrowdTraj provides pixel and real-world coordinates via per-scene homography matrices for physically meaningful analysis. Our experimental results show that tracking accuracy (IDF1) drops to 0.68 to 0.70 in the densest scenes, compared with approximately 0.90 in less crowded scenes. Trajectory prediction training also becomes substantially more computationally expensive in dense scenes, with training times increasing by up to 8 times. These findings show that CrowdTraj exposes limitations in current trajectory prediction pipelines that remain hidden on existing sparse-crowd benchmarks, particularly in robustness to tracking noise and computational scalability.
Introduction
Existing trajectory benchmarks largely omit dense, occluded CCTV crowds and often assume perfect trajectories. CrowdTraj supports end-to-end evaluation in that setting and reveals substantial performance degradation under high density and occlusion.
Introduction
- Existing benchmarks often assume perfect trajectories or feature sparse, less-occluded scenes, limiting evaluation of dense-crowd CCTV conditions.ETH-UCY and SDD average fewer than 15 pedestrians per frame; other crowd datasets described here have limitations in sequence length or viewpoint.Source: Introduction · p. 1
- CrowdTraj enables end-to-end evaluation from detection through tracking to forecasting under severe occlusion and noise across five real-world scenes.The scenes include train stations, busy pedestrian walkways, and crowd events, with a peak of 372 pedestrians in one frame.Source: Introduction · p. 1
- CrowdTraj provides real-world coordinates via homography metrics for distance-based calculations such as velocity, acceleration, and interaction radius.The benchmark contributions also include comparisons of tracking and prediction methods over 24 historical and 36 future frames.Source: Introduction · p. 1
- CrowdTraj benchmarking shows substantial accuracy and computational-efficiency degradation under high density and severe occlusion compared with sparse-crowd benchmarks.The study evaluates tracking and trajectory-prediction methods and identifies robustness to tracking noise and computational scalability as challenges.Source: Introduction · p. 2
Related Works
Prior trajectory datasets vary in crowd density, viewpoint, and tracking coverage, while trajectory prediction commonly depends on noisy tracking inputs and neighborhood data. CrowdTraj is presented as a setting for evaluating these linked challenges.
Pedestrian Trajectory Prediction dataset
- Common trajectory datasets are largely non-crowd, while dense-crowd datasets differ in viewpoint, duration, naturalness, and completeness of tracking.HT-21 uses CCTV views but lacks a long sequence; Jülich uses overhead setup scenes, and Lyon tracks only a sample outside a small exhaustive sub-region.Source: Pedestrian Trajectory Prediction dataset · p. 2
Detection-Tracking to Trajectory Predictions
- Trajectory prediction typically uses prior agent motion and neighborhood data, and tracking errors such as ID switches and fragmentation can add noise to its inputs.The passage notes that this entanglement makes crowd prediction challenging because of both heavy computation and noisy input patterns.Source: Detection-Tracking to Trajectory… · p. 2
CrowdTraj Benchmark
CrowdTraj combines head-box annotations, CCTV footage, scene-level real-world coordinates, and dataset statistics across diverse crowd settings. The section describes its collection and annotation process, quality checks, coordinate transformation, and scene characteristics.
CrowdTraj Benchmark
- Annotations use the MOT format with frame numbers, pedestrian IDs, and head-box coordinates, while images are 3840x2160.The labels include top-left x and y coordinates, width, and height.Source: CrowdTraj Benchmark · p. 2
Data Collection
- The dataset includes footage from Duri Station morning and evening scenes and its Platform 1 entrance, plus Vredeburg and the Paisley Food and Drink Festival; pedestrian faces were anonymized with DartBlur.A GoPro camera was placed at heights of 4-5 meters depending on the area; each scene was recorded for around 5 to 6 minutes, then cut to around 5,000 frames based on crowd density.Source: Data Collection · p. 3
Data Annotation
- The semiautomatic annotation process uses head detection and tracking followed by manual correction of identity switches, fragmentation, and track loss.A pedestrian disappearing under occlusion is not annotated in those frames; re-identification within 100 frames retains the identity, while longer unrecognized gaps start a new track.Source: Data Annotation · p. 3
Quality Control
- Cross-annotator head-count ambiguity averages 1.42% per frame, with P1 having the highest ambiguity because heads blend with crowd shadows.The ambiguity estimate compares manual detections on 60–100 randomly selected frames.Source: Quality Control · p. 3, Table 2 · p. 3
Real World Coordinate and Distortion
- Scene coordinates are transformed from pixels using homography, with fisheye calibration and undistortion applied to accommodate the wide-angle camera.Paisley coordinates come from Google Maps; the other listed scenes use manual real-world measurements.Source: Real World Coordinate and Distortion · pp. 3–4
Data Statistics and the Scenes
- CrowdTraj has the largest average pedestrian count, bounding-box count, and maximum density among the compared datasets.Table 1 compares dataset-level summaries and provides scene statistics for CrowdTraj; the dataset also offers pixel and real-world coordinates for location-based analysis.Source: Table 1 · p. 3, Data Statistics and the Scenes · p. 4
- The scenes span varied movement patterns, including sharp turns, a bottleneck, a three-way junction, and bidirectional festival flow.Paisley reaches 372 pedestrians in a frame and contains two independent pedestrian paths; Duri morning pedestrians have the highest velocity among its described scenes.Source: Data Statistics and the Scenes · p. 4
Algorithm Analysis
The section evaluates detection and tracking, trajectory prediction, and their combination under realistic noisy inputs, using established methods, task-specific metrics, and scene-based training and validation protocols.
Algorithm Analysis
- The evaluation separates detection-and-tracking, prediction on ground-truth tracks, and a noisy end-to-end test combining the best tracker and predictor.The combined test feeds tracker output to the best-performing prediction model.Source: Algorithm Analysis · p. 4
Baseline
- Tracking baselines pair YOLOv11 or YOLOv26 with ByteTrack, BoT-SORT, SMILETrack, or JDE.ByteTrack uses high- and low-confidence detections; BoT-SORT adds appearance cues, JDE jointly learns detection and re-identification, and SMILETrack uses learned similarity.Source: Baseline · p. 4
- Prediction baselines include SocialVAE*, MART, MoFlow, and DSTIGCN, with SocialVAE* limited to 20 closest neighbours to reduce memory use.MART builds a hypergraph per training batch; MoFlow uses flow-matching loss for diverse outputs, and DSTIGCN uses attention and a graph convolution network.Source: Baseline · pp. 4–5
Algorithm Evaluation Metrics
- Tracking is assessed with detection, association, and identity metrics including MOTA and IDF1, while prediction uses minADE and minFDE.For each agent, minADE and minFDE select the prediction with the smallest error relative to ground truth.Source: Algorithm Evaluation Metrics · p. 5
Experiments Details
- Tracking training uses the first 1,000 frames and the next 200 for validation, sampling every five frames for both detectors.Prediction uses 24 observations and 32 future steps; the 20-neighbour SocialVAE* variant avoids a full-neighbourhood graph requiring more than 200 GB of memory.Source: Experiments Details · p. 5
- Table 4 compares YOLOv26 paired with ByteTrack, BoT-SORT, or SMILETrack and YOLOv11 paired with JDE across multiple scenes.The study also reports prediction over three data splits, with training data selected to include most trajectories.Source: Experiments Details · p. 5, Table 4 · p. 6
Results
Tracking performance is strongest in less crowded scenes and degrades under severe occlusion, while prediction results vary with scene density and coordinate setting. MART leads on most pixel-coordinate scenes, whereas SocialVAE* performs better on real-world coordinates and generalizes better across most scenes.
Tracking results
- MOTA of approximately 0.81 and IDF1 of 0.90 were achieved by BoT-SORT-YOLOv26 in the less crowded Duri Morning and Duri Evening scenes.In Paisley and Duri Platform 1, high occlusion and a bottleneck respectively make tracking more difficult; SMILETrack-YOLOv26 and JDE-YOLOv11 show greater robustness, but head-only detection limits gains.Source: Results · p. 6
Trajectory prediction results
- MART’s training-time advantage reversed in Paisley, where its training time rose substantially and exceeded SocialVAE*’s.The authors attribute this to MART retaining the full neighbourhood graph in the dense scene, while SocialVAE* holds neighbourhood size constant; Paisley also has two independent pedestrian paths.Source: Results · p. 6
- Duri Morning had the worst prediction performance, with the authors linking this to its highest average pedestrian velocity under a shared prediction horizon.Faster pedestrians travel farther within that fixed horizon.Source: Results · p. 6
- MART achieved the lowest prediction error and shortest training time in 4 of 5 scenes, while SocialVAE* ranked second for accuracy.DSTIGCN and MoFlow did not achieve comparable accuracy and were prone to gradient explosion, requiring smaller batches and learning rates and an epsilon term to avoid invalid loss values.Source: Results · p. 6, Table 5 · p. 7
Real-world coordinate results
- On real-world coordinates, SocialVAE* outperformed MART in accuracy and generalization across most scenes, while both methods still required substantial training time.The authors relate this difference to MART’s joint consideration of all agents and scene-specific overfitting, versus SocialVAE*’s local neighbourhood context.Source: Table 6 · p. 7, Results · p. 7
Noisy end-to-end prediction
- With SMILETrack outputs as input, MART’s training time fluctuated across scenes, while tracking accuracy did not appear to fall significantly when training time fell.The authors attribute the fluctuations to tracking noise, including false positives and false negatives.Source: Results · p. 7
Conclusion and Future Research
CrowdTraj adds five real-world CCTV scenes with peak densities up to 372 pedestrians per frame for end-to-end evaluation under occlusion and noise. The reported results identify tracking accuracy and prediction efficiency in dense, dynamic scenes as continuing challenges.
Conclusion and Future Research
- CrowdTraj comprises five real-world CCTV scenes reaching peak densities of up to 372 pedestrians per frame, enabling end-to-end evaluation across detection, tracking, and prediction under occlusion and noise.Source: Conclusion and Future Research · p. 7
- Current tracking methods struggle with accuracy under heavy occlusion in real time, while trajectory prediction methods struggle more with computational effort.Source: Conclusion and Future Research · p. 7
- MART performs best in pixel-coordinate prediction, whereas modified SocialVAE generalizes better on real-world coordinates; efficient training in dynamic, high-density scenes remains an open challenge.The authors suggest local context plays a key role in scaling prediction to dense crowds.Source: Conclusion and Future Research · p. 7