PaperlayerView exact revision on arXiv

Source-linked AI summary

Trajectory Prediction for Autonomous Driving in Road Construction Zones Using Human-in-the-Loop Driving Data

Shuo Zhang, Aman Sunesh, Semiha Ergan, Kaan Ozbay

arXiv:2610.12747v1cs.ROcs.AIcs.LG

TL;DR

Temporary lane changes and interactions make ego-trajectory prediction in work zones challenging, and human signals may complement trajectory history. The paper collects multimodal work-zone driving data and develops WZDrive; its reported results support the value of driver-control signals and explicit work-zone context for this task.

  • Problem

    Temporary lane changes and interactions with nearby vehicles and workers make ego-trajectory prediction challenging, while trajectory history alone may not resolve prediction challenges.

    Source: §1 · p. 1

  • Method

    The paper collects human-in-the-loop work-zone driving data and develops WZDrive, combining driver-control and attention signals with trajectories, surrounding agents, lane geometry, and work-zone context.

    Source: §1 · pp. 1–2, §4.1 · p. 4

  • Results

    The reported results demonstrate the value of driver-control signals and explicit work-zone context for ego-trajectory prediction in urban roadway work zones.

    Source: §7 · p. 8

  • Takeaways & Limitations

    WZDrive-Data and its benchmark provide a foundation for studying ego-vehicle trajectory prediction in urban work zones.

    Source: §7 · p. 8

    The simulated dataset comes from 33 drivers aged 18–34, and rare events such as collisions and unexpected worker interactions remain sparsely represented.

    Source: §7 · p. 8

Abstract

from arXiv · show

Ego-trajectory prediction in urban roadway work zones remains an underexplored topic in autonomous-driving research, with limited dedicated data and systematic evaluation. Roadway work zones involve temporary changes in the lane layouts and connectivity of the roads, complicating interactions with surrounding vehicles and workers. These changes make future vehicle trajectories difficult to predict. To address these challenges, we collect WZDrive-Data, a multimodal dataset comprising 1,183 work-zone driving segments from 33 drivers, using a human-in-the-loop driving simulator with prerecorded worker animations and VR-controlled workers. Driver control and gaze signals are recorded from human participants, while vehicle trajectories and scene information are obtained from the simulator. Our proposed multimodal ego-trajectory predictor, WZDrive, achieves a mean minADE (K=6) of 1.09 m and minFDE (K=6) of 2.12 m. Compared with the best-performing state-of-the-art baseline for each metric in our benchmark, WZDrive reduces minADE (K=6) by 7.18% and minFDE (K=6) by 13.18%. This research addresses a gap in autonomous-driving research on urban work zones by combining human-in-the-loop data collection with multimodal learning from human driving signals and in the context of work zones.

1 Introduction

Temporary lane changes and interactions make work-zone ego-trajectory prediction challenging, while existing data and evaluations leave a dedicated work-zone trajectory benchmark needed. The paper introduces multimodal data collection and a predictor using human driving signals and work-zone context.

1 Introduction

  • Temporary lane changes and interactions with vehicles and workers make future motion difficult to predict in constrained work zones.The conditions can be safety-critical; the paper reports an estimated 101,000 U.S. work-zone crashes in 2023, with roughly 39,000 injuries and about 900 fatalities.Source: §1 · p. 1
  • Existing work-zone resources and general-driving benchmarks leave a need for dedicated trajectory data and systematic evaluation under work-zone conditions.ROADWork’s pathway-prediction task uses spatially sampled video-reconstructed waypoints rather than directly recorded vehicle states at fixed intervals [5].Source: §1 · p. 1
  • The paper collects WZDrive-Data from 33 participants across six simulator scenarios, combining human driving signals with vehicle trajectories and scene context.The study uses a CARLA–SUMO-based simulator with VR-controlled workers and prerecorded worker animations [13].Source: §1 · pp. 1–2
  • WZDrive combines multimodal encoding and stage-wise refinement, and the benchmark evaluates rule-based, sequence, and state-of-the-art predictors.The reported contribution is improvement over the evaluated baselines; this passage does not provide metric values.Source: §1 · p. 2

2 Related Work

Prior trajectory-prediction methods largely address general driving rather than jointly modeling temporary work-zone geometry and human-driver control signals. Existing datasets and immersive simulation provide related resources, while this work positions controlled work-zone driving data as a complement.

2.1 Trajectory Prediction

  • Trajectory-prediction methods model multimodal behavior, interactions, maps, and refinement, but primarily target general driving rather than joint work-zone geometry and driver-control signals.The reviewed approaches include map-aware, attention-based, query-based, and ego-trajectory methods.Source: §2.1 · p. 2

2.2 Driving and Roadway-Work-Zone Datasets

  • Existing driving and work-zone datasets cover trajectories, attention, visual annotation, or reconstructed paths, while WZDrive-Data pairs controlled human-driving trajectories with synchronized driver signals and explicit work-zone geometry.EPOSH provides semantic annotations, ROADWork adds annotations and reconstructed paths [5], and ROADWork4D-CL supports closed-loop planning evaluation [31].Source: §2.2 · p. 2

2.3 Human-in-the-Loop Simulation for Road-User Interaction

  • Immersive simulation supports controlled study of road-user interactions, and existing driver-centered platforms establish methods for collecting multimodal human-driving responses.Prior work includes VR pedestrian-interaction studies and platforms combining driving simulation with gaze, physiological, or vehicle signals [9] [32] [33] [37].Source: §2.3 · p. 2
  • Because work-zone layouts and environmental conditions can increase conflict potential, the paper motivates a trajectory-centered benchmark with matched control and safety-critical configurations.The benchmark combines human-driving signals with work-zone context.Source: §2.3 · p. 2

3 Dataset

WZDrive-Data is collected through a human-in-the-loop simulator with roadway-worker interaction across varied work-zone and scenario conditions. Its segments provide trajectory windows and synchronized multimodal signals for ego-trajectory prediction.

3.1 Human-in-the-Loop Data Collection

  • The collection platform integrates CARLA–SUMO co-simulation, physical driving controls, gaze tracking, scene sensing, and VR roadway-worker interaction.The study uses prerecorded worker animations in two configurations and a VR operator who places cones and interacts with the driver in another.Source: §3.1 · p. 2, Figure 1 · p. 3
  • Six randomized conditions vary warning signage, rain, road curvature, lighting, and truck involvement across three work-zone configurations.The final sample comprises 33 drivers aged 18–34; each scenario lasted approximately five minutes and included encounters with each work-zone configuration.Source: §3.1 · p. 3

3.2 Dataset Characteristics

  • WZDrive-Data contains 1,183 encounter segments from 33 drivers, covering three configurations and six scenarios, with approximately 3.7 hours of trajectory coverage and 26,892 overlapping prediction windows.Each window contains 10 observed frames and 25 future positions at 5 Hz, for a nominal 2 s history and 5 s prediction horizon.Source: §3.2 · p. 3, Table 1 · p. 4
  • The dataset combines ego motion, driver controls, gaze, surrounding agents, lane geometry, and work-zone context, alongside RGB images, semantic-segmentation images, and driving-view screenshots.Gaze and visual streams are recorded at 10 Hz, with gaze aligned to the 5 Hz trajectory sequence for model inputs.Source: §3.2 · p. 3, Table 1 · p. 4

4 Methodology

WZDrive combines vehicle motion, driver signals, and work-zone scene context to predict six candidate ego trajectories and their probabilities. Its methodology defines the prediction window, fuses multimodal inputs, and trains and refines predictions in stages.

4.1 Problem Formulation

  • Given 2 s of observations, WZDrive predicts six candidate ego trajectories over the next 5 s, represented by 25 future positions at 0.2 s intervals.Inputs include ego-motion, driver controls and gaze, surrounding vehicles and VR-controlled workers, lane geometry, and work-zone context; coordinates use a local frame centered on the latest ego position and aligned with its heading.Source: §4.1 · p. 4

4.2 Multimodal Encoding and Fusion

  • Separate encoders represent ego motion, controls, gaze, surrounding agents, lane geometry, and work-zone context; learned scoring combines motion with controls and weights relevant agent features.Individual agent features remain available for decoder attention to specific vehicles and workers.Source: §4.2 · p. 4
  • A temporary topology adapter scores lane and lane-connection suitability under current work-zone conditions, using connection position, orientation, and type to regulate lane-graph exchange; horizon-aware fusion combines four representations for 1-, 3-, and 5-s horizons.The contexts guide predictions over (0, 1], (1, 3], and (3, 5] s.Source: §4.2 · p. 4
  • The decoder predicts six 5-s trajectories using motion-prior displacements plus candidate-specific residuals, processes all candidates and future steps in one forward pass, and assigns candidate probabilities.Each trajectory contains 25 positions; temporal anchor calibration subsequently adjusts the initial trajectories.Source: §4.2 · p. 5

4.3 Stage-wise Training and Trajectory Refinement

  • Training progresses from multimodal pretraining of representations and route information to direct trajectory supervision, using reconstruction, contrastive, topology, route-coverage, travel-distance, trajectory, and endpoint objectives.The overview describes a six-stage schedule from encoder pretraining to temporal anchor calibration.Source: §4.3 · p. 5, Figure 3 · p. 5
  • After route-progress refinement and direct six-candidate prediction, AnchorCal freezes the predictor and learns longitudinal and lateral corrections at 1, 3, and 5 s, interpolating them across all 25 future steps.Only the calibration head is updated using trajectory and endpoint losses.Source: §4.3 · p. 6

5 Results

WZDrive achieves the strongest reported six-candidate results against published baselines, while ablations show that driver controls and work-zone context contribute to prediction performance. The evaluation uses participant-disjoint splits and tests generalization across additional participant partitions.

5.1 Experimental Setup

  • All models observe 2 s and predict 5 s at 5 Hz; Set1 uses 27 training drivers and participant-disjoint validation and test groups.Set1 yields 22,540/2,119/2,233 training, validation, and test windows, respectively.Source: §5.1 · p. 6

5.2 Trajectory Prediction Performance

  • Work-zone information lowers minADE for every paired baseline and minFDE for all except DeMo; MM-GRU outperforms published baselines with its selected multimodal input configuration.Among published baselines, LAFormer has the lowest minADE and DeMo the lowest minFDE.Source: §5.2 · p. 6

5.3 Ablation Studies

  • Removing driver controls causes the largest ablation degradation, raising minADE and minFDE by 17.20% and 14.17%; removing work-zone and worker context raises them by 7.54% and 12.78%.Removing gaze has little effect in this setting; Direct-K6 produces the largest error reduction, and AnchorCal further improves most metrics.Source: §5.3 · pp. 6–7, Table 4 · p. 7

5.2 Trajectory Prediction Performance

  • Across three seeds, WZDrive achieves the lowest mean errors: 1.094 ± 0.022 m minADE6 and 2.115 ± 0.082 m minFDE6; versus the best published baselines, it reduces minADE 7.18% vs LAFormer with WZ and minFDE 13.18% vs DeMo without WZ.Source: §5.2 · p. 6, Table 3 · p. 7

5.3 Ablation Studies

  • Across Set2 and Set3, error-reduction trends resemble Set1, while absolute performance differences may reflect variation in driving behavior across participant groups.Both additional partitions use participant-disjoint validation and test groups, with the remaining 27 participants in each training partition.Source: §5.3 · p. 7

6 Discussion

The discussion examines gaze and refinement behavior, illustrates varied high-error encounters, and describes possible ADAS and autonomous-driving uses. The cases underscore that abrupt or unusual human responses remain difficult to predict.

Gaze and driver behavior

  • Gaze adds limited improvement in the ablation, but visual attention remains relevant to how drivers perceive surroundings, decide, and adjust behavior in safety-critical scenarios.The five scenarios include challenges from scenario design, temporary geometry, and VR-worker interactions.Source: §6 · p. 7

Stage-wise training and trajectory refinement

  • Direct-K6 produces the largest error reduction; AnchorCal further reduces minADE6 across all three partitions and minFDE6 in Set1 and Set2, while minFDE6 rises slightly in Set3.AnchorCal adds learned position corrections at 1, 3, and 5 s, interpolated across future steps.Source: §6 · p. 7

Representative failure cases

  • The five illustrated high-error cases include collisions, unexpected stopping, cone-induced braking, and a difficult lane change that ends with the vehicle on a curb.The cases were selected from ten high-error candidates across Set1–Set3; the specific collisions, stops, and corrective actions arose from human driving rather than being prescribed.Source: §6 · pp. 7–8, Figure 4 · p. 8

Applications and pathway to deployment

  • WZDrive could support work-zone ADAS warnings and corrective interventions, or inform downstream planning and control; historical control inputs can come from the CAN bus.Gaze would be supplied separately by an eye-tracking system and could be optional depending on the driver.Source: §6 · p. 8

7 Conclusion and Future Work

The study introduces WZDrive-Data and a work-zone ego-trajectory benchmark alongside WZDrive, a multimodal predictor. Results demonstrate the value of driver-control signals and explicit work-zone context, while the authors identify data-coverage limits and frame the work as a foundation for further study.

7 Conclusion and Future Work

  • The study introduces WZDrive-Data, a work-zone ego-trajectory benchmark, and WZDrive, a multimodal predictor with stage-wise training and refinement.Source: §7 · p. 8
  • Results demonstrate the value of driver-control signals and explicit work-zone context for ego-trajectory prediction in work zones.Source: §7 · p. 8
  • The simulated data come from 33 drivers aged 18–34, so broader recruitment and real-world data are needed to assess generalization.Source: §7 · p. 8
  • Collisions, atypical stopping, and unexpected worker interactions remain sparsely represented across three work-zone configurations and six scenarios.Expanding layouts and interaction conditions could add examples for more comprehensive training and evaluation.Source: §7 · p. 8
  • The dataset, benchmark, and model provide a foundation for urban work-zone ego-trajectory research, and the collection approach could extend to vehicle-pedestrian encounters.Source: §7 · p. 8
Loading 2610.12747v1…