Source-linked AI summary
CitySim: A Drone-Based Vehicle Trajectory Dataset for Safety Oriented Research and Digital Twins
Ou Zheng, Mohamed Abdel-Aty, Lishengsa Yue, Amr Abdelraouf, Zijin Wang, Nada Mahmoud
TL;DR
Existing trajectory datasets provide limited support for simultaneously obtaining accurate trajectories and substantial safety-critical events. CitySim addresses this gap with a drone-based dataset built through accuracy-oriented processing and enriched with geometric and digital-twin assets. It reports broader safety-event coverage, including more cut-in, merge, and diverge events than HighD and NGSIM.
Problem
Existing vehicle trajectory datasets do not readily provide both high accuracy and substantial safety-critical events needed for safety-oriented research.
Method
CitySim extracts trajectories from 1,140 minutes of drone video at varied roadway locations using advanced computer vision, manual error-checking, rotated bounding boxes, and digital-twin assets.
Results
CitySim has significantly more cut-in, merge, and diverge events than HighD and NGSIM.
Takeaways & Limitations
CitySim supports safety-oriented research by combining varied vehicle interactions, geometric event representation, and digital-twin testing assets.
Takeaways & Limitations
Future work will add more interaction-rich locations, broader conflict types, traffic-flow parameters, calibrated models, and visualization tools.
Abstract
from arXiv · showhide
The development of safety-oriented research and applications requires fine-grain vehicle trajectories that not only have high accuracy, but also capture substantial safety-critical events. However, it would be challenging to satisfy both these requirements using the available vehicle trajectory datasets do not have the capacity to satisfy both.This paper introduces the CitySim dataset that has the core objective of facilitating safety-oriented research and applications. CitySim has vehicle trajectories extracted from 1140 minutes of drone videos recorded at 12 locations. It covers a variety of road geometries including freeway basic segments, signalized intersections, stop-controlled intersections, and control-free intersections. CitySim was generated through a five-step procedure that ensured trajectory accuracy. The five-step procedure included video stabilization, object filtering, multi-video stitching, object detection and tracking, and enhanced error filtering. Furthermore, CitySim provides the rotated bounding box information of a vehicle, which was demonstrated to improve safety evaluations. Compared with other video-based critical events, including cut-in, merge, and diverge events, which were validated by distributions of both minimum time-to-collision and minimum post-encroachment time. In addition, CitySim had the capability to facilitate digital-twin-related research by providing relevant assets, such as the recording locations' three-dimensional base maps and signal timings.
Dataset Generation
CitySim uses a five-step generation procedure to produce accurate vehicle trajectories. The procedure combines video processing, object detection and tracking, and enhanced error filtering.
- CitySim applies video stabilization, object filtering, multivideo stitching, detection and tracking, and enhanced error filtering.These five steps were designed to make the output trajectories as accurate as possible.
Video Stabilization
Video stabilization addresses drone motion before trajectory extraction. CitySim uses SIFT-based feature mapping, frame-based background construction, homography transformations, and CSRT backup matching.
- Drone instability from airflow and vibration makes video stabilization necessary for reliable trajectory extraction.
- SIFT features from video frames support stabilization and mapping onto a vehicle-free accumulated weighted frame.
- Homography transformations map each video frame to the accumulated weighted frame across two image planes.The paper describes homography as a transformation between planar image projections.
- CSRT provides backup feature matching when high-altitude drone movements cause SIFT mapping failures.
Object Filtering
Object filtering removes background vehicles and other objects that can trigger false detections. CitySim combines foreground/background segmentation with inpainting to clean the video background.
- Road markings and other background objects are removed because they can cause vehicle-detection false positives.
- A Gaussian-mixture-based foreground/background segmentation algorithm removes vehicles from the background before inpainting eliminates remaining undesirable objects.
Multivideo Stitching
CitySim stitches videos from multiple drones to extend trajectory coverage beyond a single drone’s limited observation area. Pairwise stitching uses color matching, SIFT features, and image blurring.
- Flight-height restrictions and video-resolution limitations restrict the observation area covered by one drone.
- Multivideo stitching combines recordings from multiple drones to increase the length of recorded vehicle trajectories.
- Each video pair is stitched using histogram color matching, SIFT features, and image blurring.Color matching unifies brightness and contrast to reduce vehicle reidentification errors across drone videos.
Object Detection and Tracking
CitySim detects vehicles with Mask R-CNN and tracks them with CSRT, while deriving rotated bounding boxes from segmentation masks to better represent vehicle geometry and movement direction.
- Object Detection and Tracking: Mask R-CNN detects vehicles by generating pixel-level segmentation masks, and CSRT tracks the detected objects.CSRT can provide an interpolated tracking location when Mask R-CNN misses a detection.
- Object Detection and Tracking: Rotated bounding boxes are derived from segmentation masks by finding the minimum rectangle aligned with the vehicle’s polygonal shape.The procedure evaluates antipodal pairs to form the polygon’s minimum rectangle.
- Object Detection and Tracking: Straight bounding boxes can overestimate vehicle size on curves or at intersections because they do not align with movement direction.Rotated boxes more accurately represent vehicle size, orientation, and location.
Enhanced Error Filtering
CitySim applies a data-fixing process to identify and correct erroneous detections and bounding boxes, with adjusted data reused in successive training rounds.
- Enhanced Error Filtering: A data-fixing tool allows users to delete wrongly detected objects and adjust erroneous bounding-box sizes and headings.Errors can arise from shadows, detection mistakes, and vehicle types absent from the training data.
Model Accuracy Evaluation
The study evaluates detection and tracking accuracy with IOU and describes CitySim’s data, roadway coverage, trajectory attributes, and safety-oriented applications.
- Model Accuracy Evaluation: IOU is used to evaluate the accuracy of the proposed detection and tracking algorithm.The supplied equation expresses IOU using the detection result and ground truth.
- Model Accuracy Evaluation: 3,541 video images from 20 randomly selected vehicles were used to calculate IOU against manually collected ground truths.The proposed algorithm improved performance across all 16 movement types, especially left- and right-turning movements.
- Dataset Coverage: CitySim contains trajectories from 1,140 minutes of drone recordings at 12 locations selected for intensive vehicle interactions and conflicts.The locations cover freeway, weaving, merge/diverge, signalized, stop-controlled, and control-free intersection settings.
- Trajectory Attributes: Each trajectory waypoint records seven vehicle key-point positions in pixels, feet, and GPS coordinates, plus speed, heading, and lane number.The heading is measured relative to both Global North and the image x-axis.
- Trajectory Examples: CitySim’s trajectory maps illustrate vehicle interactions at a weaving segment and traffic volumes from turning movements at an intersection.At Expressway A, interactions involving entering and exiting vehicles slow traffic in the delineated area.
- Safety Evaluation: Safety evaluation compares CitySim with NGSIM and HighD using freeway cut-in, merge/diverge, and intersection conflict events.Cut-in events use minimum time-to-collision, while intersection conflicts use minimum post-encroachment time.
Freeway/Expressway Cut-in/Merge/Diverge
CitySim contains substantially more freeway cut-in, merge, and diverge events than HighD and NGSIM, and its compared conflicts include more severe cases than HighD.
- Event Counts: CitySim provided larger samples of cut-in, merge, and diverge events than HighD and NGSIM for safety-oriented research.These events are heavily tested in autonomous-vehicle safety because they often involve severe conflicts or crashes.
- Event Counts: 8,873 cut-in events were observed in two CitySim locations, compared with 5,468 in HighD and 2,907 in NGSIM.The same CitySim locations contained 1,954 merge events and 3,910 diverge events, while HighD and NGSIM had very small merge and diverge samples.
- Conflict Severity: At the 5.0 s threshold, Expressway A and Freeway C had more severe cut-in and merge conflicts than HighD.The comparison considered before- and after-cut-in minTTC and excluded NGSIM from severity analysis because reported trajectory errors generated abnormal minTTCs.
Intersection Conflicts
CitySim’s rotated bounding-box trajectories reveal more and more severe critical conflicts than center-point measurements, with different intersection-event distributions across minPET thresholds. The dataset also supplies assets for digital-twin-based traffic simulation and safety research.
- Conflict measurement: Bounding-box measurements identify more critical conflict events than center-point measurements, and the detected conflicts are more severe.The paper reports that center-point analysis can miss safety-critical events or undermine their severity.
- Conflict distribution: Critical conflict-event distributions differ significantly between bounding-box and center-point trajectories at the studied intersections.Figures 16 and 17 show different spatial distributions under the compared minPET thresholds.
- Intersection A: 204 bounding-box events versus 36 center-point events were identified at Intersection A using a 1.0 s minPET threshold.At the 2.5 s threshold, the corresponding counts were 994 and 797.
- Intersection G: 32 bounding-box events versus 20 center-point events were identified at Intersection G using a 1.0 s minPET threshold.At the 2.5 s threshold, the corresponding counts were 177 and 152.
- Digital-twin applications: CitySim supports digital-twin applications by combining accurate vehicle trajectories with 3D base maps and signal timing data.The trajectories can calibrate microscopic traffic patterns for cosimulation platforms integrating microscopic and driving simulation.