INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps

• Accurate prediction and comprehensive understanding of other road users are required for fully autonomous driving in complex scenarios.Source: §I · p. 1
• Motion datasets support prediction, behavior modeling, motion representation, imitation learning, and human-like behavior generation.Source: §I · p. 1
• Public datasets such as NGSIM and highD facilitated behavior-related research but restricted it through limited diversity, complexity, criticality, map information, and interaction-entity completeness.Source: §I · pp. 1–2
• Existing public datasets are concentrated largely on highway scenarios, excluding many highly interactive settings such as roundabouts and unsignalized intersections.Source: 1) Diversity of interactive driving… · p. 2
• Datasets from different countries are needed to represent cultural differences in driving styles, preferences, risk tolerance, and traffic-rule understanding.Source: 2) International diving cultures · p. 2
• Existing scenarios are often simple and structured, with explicit right-of-way, cautious behavior, little social pressure, and few aggressive or irrational decisions.Source: 3) Complexity of the scenarios and… · p. 2

4) Criticality of the situations:

• Critical near-collision situations are sparse in existing datasets, limiting research on behavior prediction and other difficult problems.Source: 4) Criticality of the situations · p. 2

Semantic maps

• Semantic maps containing references, lanelet connections, and traffic rules provide features required for motion prediction and planning and support generalization to other scenarios.Source: 5) Map information · p. 2

6) Completeness of interaction entities:

• Complete motions of surrounding entities are needed to model, predict, and imitate interactive vehicle behavior, but onboard sensors suffer from occlusions and limited field of view.Source: 6) Completeness of interaction entities · p. 2

Dataset construction

• The proposed dataset is collected by drones and traffic cameras to emphasize international coverage, complete interactions, critical situations, and semantic maps.Source: 6) Completeness of interaction entities · p. 2

Diverse and international

• It includes diverse scenarios from different countries, including roundabouts, signalized and unsignalized intersections, and urban or highway merging and lane changes.Source: 6) Completeness of interaction entities · p. 2

Complex and critical

• The dataset contains aggressive or irrational behavior, social-pressure effects, near-collisions, slight collisions, and motions of entities that may influence driving behavior.Source: 6) Completeness of interaction entities · p. 2

Existing datasets

• NGSIM and highD provide useful vehicle-motion data, but their scenarios are limited and interactions at signalized intersections are rare.Source: A. Datasets from Bird’s Eye View · p. 2

• Onboard-sensor datasets include surrounding motions from LiDAR and cameras or many data-collection vehicles from GPS.Source: B. Datasets from Onboard Sensors · p. 3
• Onboard sensors can capture diverse, long-duration scenarios and partially recover ego-vehicle-view occlusions.Source: B. Datasets from Onboard Sensors · p. 3
• GPS-based fleets lack recordings of surrounding vehicles or pedestrians without GPS devices, making actual interactions difficult to determine.Source: B. Datasets from Onboard Sensors · p. 3
• LiDAR- and camera-based datasets cannot guarantee inclusion of all surrounding objects affecting behavior, especially when sensors have limited field of view.Source: B. Datasets from Onboard Sensors · p. 3
• Large-area data collection can produce few repetitions at the same location, making multimodal driving behavior difficult to learn for prediction or planning.Source: B. Datasets from Onboard Sensors · p. 3
• Most motion datasets lack map information, while Argoverse provides relatively rich physical and partially semantic map information.Source: B. Datasets from Onboard Sensors · p. 3
• The proposed dataset offers more diverse, complex, and critical scenarios, full-semantic HD maps, and superior interaction-entity completeness compared with three existing datasets.Source: B. Datasets from Onboard Sensors · p. 3

A. Diversity

• The dataset includes urban zipper merging, highway ramp merging and lane changes, five roundabouts, unsignalized intersections, and an unprotected left turn at a signalized intersection.Figure 2 identifies the scenario categories and their traffic-control settings.Source: A. Diversity · p. 3, Figure 2 · p. 4
• The highway scene contains both zipper merging across two lanes and forced merging across three lanes, requiring vehicles to change lanes.The upper lanes merge into one, while the lower lanes merge into two and impose lane changes.Source: A. Diversity · p. 3
• The dataset includes an extremely busy 7-way roundabout with one yield branch and six stop branches, alongside roundabouts governed entirely by yield or stop signs.Vehicles enter simultaneously at relatively high speeds, producing intensive interactions.Source: A. Diversity · pp. 3–4
• It also contains busy all-way-stop intersections where multiple vehicles compete by inching forward, plus intersections combining stop-controlled branches with right-of-way traffic.The examples include a 9-lane all-way-stop intersection and a busy all-way-stop T-intersection.Source: A. Diversity · p. 4

B. Internationality

• Motion data spans three continents and four countries— the US, China, Germany, and Bulgaria—while all included countries drive on the right.The paper notes remarkable distinctions in driving culture despite this shared traffic-side convention.Source: B. Internationality · pp. 4–5
• Comparable roundabouts from the US, Germany, and China share yield-based nominal rules, while German and Chinese zipper-merge scenarios retain similar rules and heavy-traffic speeds.The roundabout examples lack stop signs, and the zipper scenarios remain comparable despite differing road contexts.Source: B. Internationality · p. 5

C. Complexity

• Strongly interactive vehicle pairs can appear every few seconds in selected ramps, entrances, and stop-controlled intersections.Examples include the ramp in ZS, entrance branches in FT, all-way-stop intersections in EP and MA, and a two-way-stop intersection in GL.Source: C. Complexity · p. 5
• Unstructured roads without explicit lane restrictions enable irrational and dangerous insertions, while drivers negotiate near-simultaneous arrivals through inching or acceleration.In GL, a vehicle inserted between two stopped vehicles despite lacking explicit road structure; in MA and EP, drivers negotiated around ambiguous right-of-way.Source: C. Complexity · p. 5
• Right-of-way violations create adversarial interactions, including a vehicle entering a roundabout and forcing the vehicle with right-of-way to stop and yield.The example occurs in FT, where V0 violated the rule while V1 was already in the roundabout.Source: C. Complexity · p. 5
• Social pressure and delayed entry for vehicles without nominal right-of-way further increase behavioral complexity.Vehicles may wait minutes to enter and pass, with queues and honking contributing to impatience and aggressive behavior.Source: C. Complexity · p. 5
• Critical interactions include extremely low time-to-collision-point situations, emergency swerves, and slight collisions.Examples include a near-collision in GL requiring an emergency swerve and a slight collision involving conflicting right-turn predictions.Source: D. Criticality · pp. 5–6

E. Semantic Map

• The semantic maps include physical layers, reference paths, lanelets and their connections, turn directions, traffic rules, and right-of-way information.The information is organized in a consistent format and toolkit for dataset users.Source: E. Semantic Map · p. 6
• The construction pipeline combines drone video stabilization and map alignment with object detection, tracking, and trajectory smoothing.Raw 4K video at 30 Hz is downsampled to 10 Hz; Faster R-CNN, Kalman filtering, and an RTS smoother are used in processing.Source: §IV · p. 6, A. Motions from Drone Data · p. 6
• Lanelet2 maps encode road borders, lane markings, traffic signs, lane courses, and regulatory elements such as right-of-way and speed limits.The physical layer is used to create atomic lanelets, which form the basis for regulatory elements.Source: Figure 7 · p. 6

B. Motions from Traffic Camera Data

• Traffic-camera processing detects vehicles and pedestrians using 2D bounding boxes, instance masks, and instance types.The pipeline begins with a state-of-the-art object detector applied to each frame.Source: B. Motions from Traffic Camera Data · p. 6
• Detections are associated into tracks using mask-overlap tracking combined with a visual tracker that compensates for missed detections.The association combines an Intersection-over-Union tracker with a visual tracker.Source: B. Motions from Traffic Camera Data · p. 6
• Ground-plane trajectories are estimated and smoothed with a pin-hole camera observation model, uncertainty handling, and a bicycle process model for vehicles.The bicycle model captures vehicle kinematic constraints while incorporating pixel-level measurement uncertainty.Source: B. Motions from Traffic Camera Data · p. 6
• Lanelet2 maps provide centimeter-accurate representations of road geometry and traffic regulations for trajectory analysis.The maps encode road borders, lane markings, traffic signs, lanelets, and regulatory elements.Source: C. Construction of the High Definition… · pp. 6–7
• Combining trajectories with maps helps explain deceleration near junctions based on right-of-way and interacting traffic participants.The maps support reasoning about why vehicles respond differently to the same junction layout.Source: C. Construction of the High Definition… · p. 7
• The dataset covers roundabouts, unsignalized and signalized intersections, merging, and lane-change scenarios.The paper reports vehicle trajectories across multiple locations and scenario categories.Source: A. Scenarios and Vehicle Density · p. 7

• IPV counts interaction pairs per vehicle using rule-based extraction across different spatial representations of vehicle paths.The metric is used to represent interactive-behavior density.Source: B. Metrics for Interactive Behavior… · p. 7
• ΔTTCPmin measures relative vehicle states when paths share a conflict point without a forced stop.It covers both static crossing or merging points and dynamic merging points derived from actual trajectories.Source: B. Metrics for Interactive Behavior… · p. 7, Figure 8 · p. 8
• TTCPmin ≤3 s defines an interaction, with traveling time computed from each vehicle’s speed and distance to the conflict point.The interaction period determines the start and end times used in the metric.Source: B. Metrics for Interactive Behavior… · p. 7

C. Distribution of Interactivity

• 13,375 interactive vehicle pairs were identified, and INTERACTION contains more intensive interactions with ΔTTCPmin ≤1 s than highD and NGSIM.The comparison plots vehicle counts and normalized density against ΔTTCPmin.Source: C. Distribution of Interactivity · p. 7
• Across scenarios, the dataset has high vehicle density at ΔTTCPmin ≤1 s and waiting periods greater than 3 s.These distributions summarize interactive trajectories over different driving scenarios.Source: C. Distribution of Interactivity · p. 8

• The dataset provides utilization examples spanning motion prediction, imitation learning, planning, clustering, interaction extraction, and human-like behavior generation.These examples illustrate the dataset’s intended research applications.Source: §VI · p. 8
• High-density interactive trajectories and HD semantic maps support both learning-based and planning-based motion prediction approaches.The maps and trajectories are described as useful for prediction in intensive-interaction situations.Source: A. Motion Prediction and Behavior… · p. 8

Motion/trajectory prediction

• A WAE-based prediction method outperformed VAE, auto-encoder, and GAN models on position and yaw-angle RMSE and MAE in the reported comparison.The experiment used motion data from the FT scenario.Source: A. Motion Prediction and Behavior… · p. 8
• The utilization section reports prediction-accuracy comparisons from method [36].Source: Table III · p. 8
• HD semantic maps enabled a CVAE and inverse-reinforcement-learning planner to predict both irrational and rational vehicle behavior.The method defined learning features in the Frenet frame and reported improved generalization.Source: A. Motion Prediction and Behavior… · p. 8

B. Imitation Learning

• Imitation learning uses semantic HD maps and surrounding-vehicle states as features to imitate human driving in the FT roundabout scenario.The framework extends the fast integrated learning and control method from.Source: B. Imitation Learning · p. 8, Figure 12 · p. 10

C. Validation of Decision and Planning

• Dataset replay motions are suitable for testing planners when surrounding entities are independent of the ego vehicle’s motions.Examples include the ego vehicle lacking right-of-way or others violating traffic rules.Source: C. Validation of Decision and Planning · p. 8
• A planner tested in the FT roundabout decelerated to avoid a collision with a vehicle entering despite the autonomous vehicle having right-of-way.The result is shown as a bird’s-eye-view simulation screenshot.Source: C. Validation of Decision and Planning · pp. 8, 10, Figure 13 · p. 10
• A DBN-based predictor returned P(exit)=0.626 when another roundabout vehicle’s intention was unclear, supporting uncertainty-aware decision and planning.The framework combined integrated decision-making, sample-based planning, and probabilistic intention prediction.Source: C. Validation of Decision and Planning · p. 10
• Long-term planning for the yielding case guaranteed a full stop in the worst case, while the non-conservatively defensive strategy preserved planned acceleration when appropriate.The reported speed profiles correspond to screenshots of the FT roundabout situations.Source: C. Validation of Decision and Planning · p. 10, Figure 14 · p. 10

D. Motion Clustering and Representation Learning

• X-means clustered trajectories using vehicle-motion features represented in the Frenet frame from map information.The resulting clusters are visualized with the map and interacting vehicles’ longitudinal positions and speeds.Source: D. Motion Clustering and… · p. 10, Figure 15 · p. 11
• Different interactive motions were separated while similar motions were clustered, supporting extraction of motion patterns.The results are shown in trajectory, state-coordinate, and PCA feature-space views.Source: D. Motion Clustering and… · p. 11

Interaction extraction

• The dataset was used to learn interaction relationships between pairs of agents through an interaction-frame extraction method.Examples include interacting car pairs in the FT scenario.Source: E. Extraction of Interactive Agents… · p. 11

F. Human-like Decision and Behavior Generation

• A CPT-based human-behavior model outperformed a TTCP model and matched a neural-network model with less training data and better interpretability.All three models were learned and tested using FT roundabout data.Source: F. Human-like Decision and Behavior… · p. 11

VII. CONCLUSION

• The dataset’s diverse, critical, and semantically mapped scenarios support representation learning, interaction extraction, and human-like behavior generation.The conclusion also lists prediction, imitation, decision-making, and planning as supported research areas.Source: §VII · p. 11