Source-linked AI summary
The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems
Robert Krajewski, Julian Bock, Laurent Kloeker, Lutz Eckstein
TL;DR
Existing measurement methods do not consistently provide the naturalistic, sufficiently detailed, and efficiently collected data needed for scenario-based safety validation of highly automated vehicles. The paper proposes drone-based aerial measurement and presents highD, a large German-highway trajectory dataset whose extracted maneuvers and traffic statistics support safety-validation analysis.
Problem
Existing measurement methods fail to consistently provide naturalistic behavior and sufficiently complete, accurate scenario descriptions for highly automated-driving safety validation.
Method
The paper uses camera-equipped drones and computer-vision algorithms to measure vehicle trajectories from an aerial perspective and construct the highD dataset.
Results
16.5 h of video produced more than 45 000 km of naturalistic driving behavior, with highD supporting extraction of typical maneuvers and inter- and intra-maneuver probabilities.
Takeaways & Limitations
highD is presented as an appropriate data source for safety validation and as a resource for research using naturalistic traffic trajectory data.
Abstract
from arXiv · showhide
Scenario-based testing for the safety validation of highly automated vehicles is a promising approach that is being examined in research and industry. This approach heavily relies on data from real-world scenarios to derive the necessary scenario information for testing. Measurement data should be collected at a reasonable effort, contain naturalistic behavior of road users and include all data relevant for a description of the identified scenarios in sufficient quality. However, the current measurement methods fail to meet at least one of the requirements. Thus, we propose a novel method to measure data from an aerial perspective for scenario-based validation fulfilling the mentioned requirements. Furthermore, we provide a large-scale naturalistic vehicle trajectory dataset from German highways called highD. We evaluate the data in terms of quantity, variety and contained scenarios. Our dataset consists of 16.5 hours of measurements from six locations with 110 000 vehicles, a total driven distance of 45 000 km and 5600 recorded complete lane changes. The highD dataset is available online at: http://www.highD-dataset.com
I. INTRODUCTION
Scenario-based validation of highly automated driving requires detailed, naturalistic real-world traffic data, but existing measurement methods have important limitations. The paper proposes aerial drone measurements and introduces the highD highway trajectory dataset.
- Measurement gap: Existing safety-validation methods include driving tests, naturalistic driving studies, field operational tests, pilot studies, and roadside sensors.These methods do not always provide sufficient dynamic scenario quality or naturalistic behavior because of sensor limitations and visibility.
- Proposed approach: Camera-equipped drones measure every vehicle’s position and movements from a bird’s-eye view for scenario-based validation.The aerial perspective improves longitudinal and lateral accuracy, avoids vehicle occlusion, and is unlikely to influence passing drivers.
- Proposed approach: The recordings cover a German-highway road segment of about 420 m from a hovering drone.The setup captures traffic from above while sacrificing object-height information, which can be estimated from object type.
- Contributions: The paper demonstrates feasibility, analyzes the methods, and provides the highD dataset of naturalistic vehicle trajectories on German highways.highD is intended for safety validation and impact assessment while also supporting traffic simulation, traffic analysis, driver modeling, and prediction research.
II. PREVIOUS WORK
Prior datasets and measurement studies provide useful traffic data but leave a gap for highway vehicle trajectories collected from aerial video and suitable for highly automated-vehicle safety validation.
- Drone-based monitoring: Earlier drone-based traffic studies primarily extracted macroscopic measures such as density, flow, and speed rather than decimeter-accurate road-user trajectories.Their trajectories were therefore not suitable for safety validation of highly automated vehicles.
- Existing trajectory datasets: The Stanford Drone Dataset contains about 17 hours of multi-road-user trajectories, but it focuses on pedestrian behavior and includes few cars at most locations.Its recordings were made on the Stanford Campus and do not contain public roads.
- Research gap: No public highway vehicle-trajectory dataset created with drone video had been identified, and aerial-video applicability to highly automated-vehicle safety validation had not yet been shown.This gap motivates the highD dataset and its validation-oriented use case.
- Onboard datasets: EuroFOT collected more than 35 million driven kilometers from around 1,200 drivers using onboard vehicle sensors.Its recordings include CAN-bus, raw video, GPS, radar, and camera data.
- Existing trajectory datasets: KITTI and Cityscapes mainly focus on urban scenes, whereas NGSIM focuses on highway and urban vehicle trajectories from elevated viewpoints.NGSIM is consequently the dataset most similar to highD and is analyzed in greater detail.
C. NGSIM Dataset
NGSIM is a major naturalistic vehicle-trajectory dataset collected from elevated video, but its raw tracks require correction before analysis. The paper frames measurement requirements around naturalistic behavior, scenario completeness, effort, and flexibility.
- NGSIM Dataset: NGSIM is widely used for traffic-flow and driver-model research as a large naturalistic vehicle-trajectory dataset.Its recordings include highway sites comparable to the German Autobahn and urban scenes.
- NGSIM limitations: Raw NGSIM trajectories contain false-positive collisions and physically illogical speeds and accelerations.Previous work refined some longitudinal movements, but other cases required manual re-extraction from recordings.
- Measurement requirements: Safety-validation measurements should capture relevant traffic facets with sufficient accuracy across static, dynamic, environmental, and behavioral dimensions.The requirements include naturalistic behavior, static and dynamic scenario descriptions, effort effectiveness, and flexibility.
- Measurement requirements: Effort effectiveness is defined as measured scenarios divided by combined permanent and initial measurement effort.Initial effort covers setup, while permanent effort covers operation.
B. Comparison of Measurement Methods
The aerial perspective preserves naturalistic behavior and provides broad, high-quality scenario descriptions, while sacrificing some flexibility and effort effectiveness. It also offers data-privacy advantages but requires substantial processing resources for online use.
- Aerial measurement preserves naturalistic behavior best because road users are unaware of the recording.
- Static scenario information can be derived from digital maps and extracted directly from aerial recordings at fixed locations.
- Aerial measurement provides high-quality dynamic descriptions because all vehicles on every lane and direction remain visible with constant high accuracy.
- Aerial measurement is less effort-effective than onboard measurement with series-production sensors because flight approval, piloting, and travel are required.
- Aerial measurement is less flexible than vehicle-based measurement because flight restrictions and environmental conditions limit recordings to daytime and calm weather.
- Aerial measurement protects privacy because high-altitude recordings do not identify road users, but accurate online processing requires high-resolution video and strong computational resources.
IV. THE HIGHD DATASET COLLECTION PIPELINE
highD is a large-scale dataset of post-processed vehicle trajectories recorded by drones at six German highway locations around Cologne. The dataset and supporting Matlab and Python tools are publicly available.
- 110 000 vehicles were extracted from 60 drone recordings totaling 16.5 hours at six German highway locations.The recordings cover approximately 420 m road segments, and each vehicle is visible for a median 13.6 s.
- The dataset contains cars and trucks recorded around Cologne during 2017 and 2018.
- Vehicle trajectories were automatically extracted with computer vision algorithms, while infrastructure was annotated manually.
- Matlab and Python source code for handling data, creating visualizations, and extracting maneuvers is provided with the downloadable dataset.
B. Video Recordings
The recordings used 4K drone video stabilized and rotated to produce a consistent highway view. Vehicle annotation was automated with semantic segmentation because the dataset contained more than 110 000 vehicles.
- 4K video was recorded at 25 fps with a DJI Phantom 4 Pro Plus positioned beside German highways.
- The drone location minimized perspective distortion and reduced the visible vehicle sidewalls.
- Videos were stabilized by estimating background transformations, then rotated so lane markings were horizontal.
- An adapted U-Net segmented vehicle pixels from background pixels, and pixel clusters were converted into vehicle bounding boxes.
- Static objects including lane markings, traffic signs, and speed limits were annotated manually because this required little effort relative to vehicle annotation.
D. Track Postprocessing
The pipeline links frame-level vehicle detections into tracks, smooths their trajectories, and derives predefined maneuvers and neighboring-vehicle measures. Recordings were collected at highway locations around Cologne.
- Track formation: Detections in consecutive frames were matched by distance to form vehicle tracks, while infeasible matches were discarded.False positives could be completely removed, and short detection gaps were bridged by predicting vehicle movements.
- Trajectory refinement: RTS smoothing with a constant-acceleration model refined positions, speeds, and accelerations in both x- and y-direction.The refinement improved positioning error to pixel-size level.
- Maneuver classification: Predefined rules and thresholds identify maneuvers such as free driving, vehicle following, critical maneuvers, and lane changes.The maneuver definitions are not mutually exclusive except for free driving and vehicle following.
- Recording locations: Recordings were made at six highway locations around Cologne selected by typical traffic density and number of lanes.
- Neighbor relations: The dataset derives ID, DHW, THW, and TTC measures for preceding and following vehicles on the own and adjacent lanes.
F. Dataset Format
highD provides substantially more and more varied highway trajectory data than NGSIM, with detailed per-track and per-frame information supporting traffic analysis.
- F. Dataset Format: Each recording supplies four files: a site description, vehicle summaries, extracted trajectories, and an aerial view of the highway area.The files describe infrastructure, vehicle dimensions and classes, speeds, accelerations, lane positions, and surrounding vehicles.
- A. General and Size Comparison of the Datasets: More than 16.5 hours of recordings from six sites are included in highD, compared with about 90 minutes from two NGSIM sites.The sites cover German highways with two or three lanes per direction, whereas NGSIM sites have five or six.
- A. General and Size Comparison of the Datasets: Nearly twelve times as many vehicles and nine times the traveled distance are recorded in highD compared with NGSIM.HighD contains 23% trucks versus 3% in NGSIM, while total vehicle travel time is almost three times as long.
- B. Variety of Included Data: HighD offers greater data variety through more recording sites, more recording times, and a broader range of mean track speeds.The speed distribution includes peaks at 80 km/h and 120 km/h, associated with trucks and cars.
- B. Variety of Included Data: The comparison table organizes the available data amounts for NGSIM and highD.Its comparison concerns recording duration, sites, vehicles, travel distance, and vehicle composition as described in the surrounding text.
- B. Variety of Included Data: Truck ratios vary from 0% to more than 50% over time in highD, while remaining below 10% in NGSIM.This indicates wider variation in vehicle-type composition across highD recordings.
C. Quality Evaluation and Comparison
The quality evaluation contrasts highD’s automated detection and post-processing with substantial tracking errors in raw NGSIM data, while also examining dataset variety and lane-change occurrences.
- Detection and Tracking Quality: About 3000 training patches were augmented to 12 000 vehicles for semantic-segmentation-based vehicle detection.The thresholds favor a low false-negative rate to detect most vehicles.
- Detection and Tracking Quality: NGSIM bounding boxes rarely match vehicle shapes, with some outliers containing mostly road surface because of tracking and image-stitching errors.These errors are especially associated with transitions between cameras and necessary rectification.
- Detection and Tracking Quality: Raw NGSIM trajectories contain false-positive collisions and physically illogical speeds and accelerations that require refinement or manual re-extraction.Filtering and longitudinal refinement are insufficient for every case.
- Detection and Tracking Quality: HighD requires no further trajectory post-processing because false-positive detections are removed and extracted trajectories are smoothed.The dataset benefits from a single high-resolution camera, a frame rate more than twice as high, and a state-of-the-art detector.
- Maneuver Statistics: More than 11 000 lane changes occur in highD, including 5600 completely performed within the observed area.HighD has twice as many lane changes as NGSIM but a lower rate per vehicle, 0.10 versus 0.45.
VI. ANALYSIS OF EXTRACTED LANE CHANGES
The lane-change analysis parameterizes trajectories with a constrained polynomial model and optimizes its parameters to represent observed maneuvers.
- A. Lane Change Trajectory Model: Lane changes are modeled with separate quadratic longitudinal and fifth-degree lateral polynomials.The model assumes zero lateral and longitudinal acceleration and zero lateral speed at the maneuver’s beginning and end.
- A. Lane Change Trajectory Model: Five remaining degrees of freedom describe each lane change using intuitive geometric, speed, and duration parameters.These include distance to the crossed lane marking, beginning and ending longitudinal speed, and lane-change duration.
- A. Lane Change Trajectory Model: Lane changes are detected through lane crossings, and lateral movement determines the maneuver’s beginning and end.The model parameters are then fitted by solving an optimization problem.
B. Description of Lane Change Surroundings
The analysis characterizes lane changes using surrounding-vehicle interactions and extracts cut-in scenarios from the perspective of a vehicle following in the destination lane.
- B. Description of Lane Change Surroundings: Statistics include the preceding vehicle on the original lane and the preceding and tailing vehicles on the destination lane.For these vehicles, the analysis extracts minimal DHW, THW, TTC, and gap size.
- B. Description of Lane Change Surroundings: 850 right-side cut-in scenarios were extracted from 5600 parameterized highD lane changes.The tailing vehicle is treated as automated and the lane change as a cut-in requiring a possible reaction.
- B. Description of Lane Change Surroundings: HighD supports safety-validation statistics by combining maneuver parameters with surrounding-vehicle conditions and their distributions.Parameter combinations indicate the circumstances under which lane changes occur.
- B. Description of Lane Change Surroundings: Figure 7 relates the tailing vehicle’s THW at lane entry to the tailing vehicle’s speed, showing the median and deciles.The statistic describes the temporal gap experienced by the following vehicle during a cut-in.