Source-linked AI summary
Revisiting Direct Control: A Field Evaluation of Sensor Fusion Efficacy in Teleoperating Full-Scale Autonomous Vehicles via Bird's Eye View
Stanisław Kuczma, Sławomir K. Tadeja, Michał R. Nowicki
TL;DR
Prior teleoperation studies often use low-fidelity settings that do not capture full-scale vehicle dynamics. This paper compares visual representations in a field experiment with human operators driving a real vehicle, finding benefits for the fused interface over single-sensor views.
Problem
Many interface evaluations use low-fidelity environments that do not reproduce full-scale vehicle dynamics, leaving limited evidence about BEV representations for human operators.
Source: §I · p. 1, D. Practical Evaluation and Fidelity… · p. 3Method
The study compares visual representations, including Camera-LiDAR BEV fusion, in a field experiment with human operators driving a real vehicle.
Source: §I · p. 2Results
The fused interface reduced task completion time relative to LiDAR-only BEV and had superior SUS and lower cognitive workload scores than single-sensor setups; SUS was 77.9/100.
Source: §VI · p. 7, §VII · p. 8Takeaways & Limitations
The fused BEV interface synchronizes depth precision with semantic clarity while reducing operator mental workload and maximizing safety margins.
Source: §VI · p. 7Total end-to-end system latency, including software and display delays, was not formally quantified.
Source: A. System Architecture and Apparatus · p. 4
Abstract
from arXiv · showhide
In autonomous driving, robot teleoperation serves as a critical safety fallback mechanism, requiring operators to assume direct control in the event of system failure. While recent research has pivoted toward high-level supervisory control or simulation-based studies, these approaches often fail to capture the physical dynamics and sensory demands of driving a full-scale vehicle. To address this gap, we evaluated the impact of spatial data representation on operator performance using a real car-sized vehicle in a field experiment with $N = 18$ participants. We compared three visualization interfaces: (i) a standard camera feed baseline, with two bird's-eye views (BEV) from (ii) a raw LiDAR point cloud, and (iii) a sensor fusion of LiDAR and camera data streams. Contrary to prior passive observation studies suggesting that sensor-fusion BEV may hinder awareness, our active-control results demonstrate that such representation significantly outperforms the standard LiDAR BEV in perceived usability ($p \leq 0.0001$) and cognitive task load ($p = 0.01$, NASA-TLX score: 34.58/100) while also being significantly better than the Camera-only view ($p = 0.043$). The fused interface also led to statistically significant reductions in task completion time ($p = 0.0461$) and superior usability (SUS: 77.9/100) compared to those of the Camera-only and LiDAR-only views. By synthesizing objective performance data with comprehensive subjective assessments, we demonstrate that BEV fusion maximizes safety margins while minimizing the operator's mental demand. These findings challenge existing teleoperation conventions and provide an empirically validated approach for designing robust remote piloting systems.
I. INTRODUCTION
Remote teleoperation must bridge operators’ sensory disconnection while avoiding information overload, yet much prior evaluation used low-fidelity settings. This study compares camera and bird’s-eye-view representations in a full-scale vehicle field experiment.
I. INTRODUCTION
- Remote operators lack vehicle-force and sound cues, while unclear or excessive visual information can erode trust and increase errors.The paper identifies visualization as a means to bridge this sensory gap and support situational awareness.Source: §I · p. 1
- Many interface evaluations use prototypes, miniature platforms, or simulations that do not reproduce full-scale vehicle dynamics.Prior work also characterized BEV as potentially inferior or overwhelming for human operators compared with camera feeds.Source: §I · p. 1
- The field experiment compared perspective video and Camera-LiDAR BEV fusion on a full-scale Pixloop vehicle with 18 human operators.The study aimed to determine which representation best optimized performance under real vehicle dynamics.Source: §I · p. 2
- The paper contributes a modular Camera-LiDAR BEV interface and a field evaluation reporting reduced workload and risk in full-scale operations.The introduction frames the evaluation as testing whether findings from low-fidelity setups translate to real-world dynamics.Source: §I · p. 2
II. RELATED WORK AND MOTIVATION
The related-work review organizes teleoperation research by sensory representation and evaluation fidelity. Camera feeds are intuitive and widely available, but have depth and viewing limitations that spatial perspectives can partly address.
II. RELATED WORK AND MOTIVATION
- The review groups remote-vehicle research by sensory representation and the fidelity of its evaluation environment.It identifies situational awareness as a continuing challenge for remote operators.Source: §II · p. 2
A. Camera-Based Feed
- Camera feeds are low-cost, widely available, and intuitive because they resemble human vision.They are commonly used as forward-facing views, sometimes supplemented with side cameras.Source: A. Camera-Based Feed · p. 2
- Camera performance can degrade in darkness or glare, and direct video provides limited depth for estimating obstacle distance.Multi-camera feeds can also introduce edge distortion and stitching inconsistencies.Source: A. Camera-Based Feed · p. 2
- Projecting camera feeds onto a spatial canvas enables third-person views that can improve trajectory stability and spatial awareness.A narrow camera field of view remains a limitation and motivates integrating additional sensors.Source: A. Camera-Based Feed · p. 2
B. LiDAR-Based Feed
Third-person views and trajectory overlays are common teleoperation design elements, while sensor fusion offers a unified spatial context but can also burden perception. The paper motivates testing whether BEV’s theoretical benefits transfer to active, real-world driving.
B. LiDAR-Based Feed
- Third-person views outperform first-person views in teleoperation research, particularly when enriched with 3D LiDAR data.These views often include a vehicle model to clarify its position relative to obstacles, though much evaluation uses miniature platforms and unlimited bandwidth.Source: B. LiDAR-Based Feed · p. 3
- Trajectory overlays are an established interface element reported to improve operator performance and confidence.They appear across interfaces using camera, LiDAR, or augmented reconstructions.Source: B. LiDAR-Based Feed · p. 3
C. Multi-Sensory Feeds Fusion
- Sensor fusion combines sensing modalities, but earlier work reported increased workload and reduced performance while later systems emphasized redundant, complementary sensing.Examples include systems providing operators with camera, LiDAR, and radar data.Source: C. Multi-Sensory Feeds Fusion · p. 3
- BEV offers a unified spatial context for sensor fusion, but can hinder detection of essential features and more information can increase overload risk.The paper asks whether BEV’s theoretical benefits translate to real-world driving under direct control rather than passive observation.Source: C. Multi-Sensory Feeds Fusion · p. 3
D. Practical Evaluation and Fidelity Challenges
The paper identifies a gap between increasingly sophisticated interfaces and their validation under full-scale vehicle dynamics. It evaluates its interface on a real vehicle, while noting that sensor placement and latency measurement constrain the setup.
D. Practical Evaluation and Fidelity Challenges
- Prototypes and miniature platforms often omit full-scale motion inertia and sensory disconnection, while some full-size studies omit formal human-subject evaluation.These constraints motivate evaluation under authentic vehicle dynamics.Source: D. Practical Evaluation and Fidelity… · p. 3
- The study uses a full-scale Pixloop vehicle and 18 participants to evaluate whether Camera-LiDAR BEV fusion improves performance and reduces workload under authentic dynamics.The paper presents this as a direct empirical test against conclusions drawn from prior work.Source: D. Practical Evaluation and Fidelity… · p. 3
A. System Architecture and Apparatus
- The test platform combines three cameras with a 360-degree LiDAR, while RTK GNSS provides a ground-truth trajectory reference.The cameras capture 960x600-pixel images at 15 frames per second; the LiDAR provides 128 vertical layers at 10 Hz.Source: A. System Architecture and Apparatus · p. 3
- Video and control commands use separate communication streams, but total end-to-end latency was not formally quantified.The video transmission latency did not exceed 25 ms; sensor placement was optimized for autonomous navigation rather than teleoperation.Source: A. System Architecture and Apparatus · pp. 3–4
B. Data Representation Layouts
The three layouts pair a camera feed with either camera imagery alone, a LiDAR point-cloud BEV, or a fused camera-and-LiDAR BEV. The fusion layout is intended to connect abstract LiDAR data with an intuitive camera view.
B. Data Representation Layouts
- The Camera layout uses a front-facing camera with additional side cameras and steering-angle-based predicted-motion cues.The multiple views help estimate distances to nearby obstacles.Source: B. Data Representation Layouts · p. 4
- The LiDAR BEV uses a filtered point cloud in an isometric view, with color grading indicating distances from 1 to 20 meters.The ground filter removes ground-level points, and the virtual camera is elevated 7 meters and angled downward 30 degrees.Source: B. Data Representation Layouts · p. 4
- The Camera+LiDAR BEV combines a front-facing video stream with a top-down LiDAR view to link abstract point-cloud data to the more intuitive camera feed.The layout was designed to provide a global overview supporting spatial orientation and obstacle localization.Source: B. Data Representation Layouts · p. 4
IV. USER STUDY
The user study compared three sensor-data presentations for full-scale vehicle teleoperation, assessing objective performance and subjective operator experience. Its hypotheses concerned whether sensor type affects usability, taskload, and teleoperation performance.
IV. USER STUDY
- A within-subject study compared Camera, LiDAR, and Camera+LiDAR presentations for full-scale vehicle teleoperation.The protocol received ethics approval and used a conservative α = 0.05 significance threshold.Source: §IV · p. 4, A. Hypotheses · pp. 4–5
A. Hypotheses
- The user-experience hypotheses predicted differences across sensor types in usability and cognitive taskload.The null hypothesis stated there was no sensor-type effect.Source: A. Hypotheses · p. 5
- The performance hypotheses tested whether teleoperation performance varies by sensor type.The null hypothesis stated performance was independent of sensor type.Source: A. Hypotheses · p. 5
B. Participants
The field study recruited 18 participants and used a counterbalanced, repeated obstacle-course task to compare the three visualizations. The course tasks and collected measures assessed maneuvering precision, spatial awareness, and stopping accuracy.
B. Participants
- 18 opportunistically sampled participants took part; most were university students, and 13/18 reported little or no teleoperation experience.All but one held a valid driver’s license, and participants reported good or corrected-to-normal vision.Source: B. Participants · p. 5
C. Experimental Design and Task
- The approximately 40-by-10-meter course included precision parking, a tight turn, obstacle avoidance, and precision stopping.These tasks were designed to assess control precision, spatial awareness, and distance estimation.Source: C. Experimental Design and Task · p. 5, Figure 4 · p. 5, Figure 5 · p. 5
- The within-participant experiment used three course laps, one per visualization, with interface order counterbalanced using a Latin square design.Course direction was split clockwise versus anticlockwise to control directional bias; participants completed SUS and NASA-TLX after each lap.Source: C. Experimental Design and Task · p. 5
- Performance measures included lap time, obstacle collisions, stop-line and parking-position distances, and trajectory error against a recorded GNSS reference path.Trajectory error was defined as average distance from actual GNSS coordinates to nearest points on the reference path.Source: C. Experimental Design and Task · p. 6
A. Perceived Usability and Taskload
Camera+LiDAR scored highest for perceived usability and lowest for workload, with significant advantages over LiDAR on both measures. Its usability also exceeded Camera’s, while the workload comparison against Camera was not statistically significant.
A. Perceived Usability and Taskload
- 77.9/100 was the Camera+LiDAR mean SUS score, above the 68 acceptability threshold; it was significantly higher than both LiDAR (p ≤ 0.0001) and Camera (p = 0.043).The Camera score did not cross the acceptability threshold, while LiDAR fell below it.Source: A. Perceived Usability and Taskload · p. 6, Table II · p. 6, Figure 6 · p. 6
- The NASA-TLX ANOVA found an interface-type effect on workload, F(2, 17) = 4.626, p = 0.014, η2 = 0.154.Post-hoc tests identified the significant pairwise difference between Camera+LiDAR and LiDAR.Source: A. Perceived Usability and Taskload · p. 6
- 34.58 was the Camera+LiDAR mean NASA-TLX score, the lowest workload of the three interfaces; its workload was significantly lower than LiDAR’s (p ≤ 0.01).The Camera+LiDAR versus LiDAR mean difference was −13.38; other pairwise workload comparisons were not significant.Source: A. Perceived Usability and Taskload · p. 6, Table II · p. 6, Figure 7 · p. 6
- The study rejected H01 and supported HA1: usability and cognitive taskload differed across interfaces, with Camera+LiDAR outperforming LiDAR on both measures.Camera+LiDAR also had higher mean SUS than Camera, whose mean did not reach the stated acceptability threshold.Source: A. Perceived Usability and Taskload · pp. 6–7
B. Teleoperation Performance
Camera+LiDAR BEV produced shorter completion times than LiDAR-only BEV, while collision comparisons showed no significant differences. The analysis used different tests according to outcome distributions and data type.
B. Teleoperation Performance
- Collision comparisons were not statistically significant after collision data were analyzed separately using GLM regression.Approximately 8.54% of data were missing, and IQR-based processing removed approximately 7.42% of anomalous data points.Source: B. Teleoperation Performance · p. 7
- Parking was analyzed with one-way ANOVA, while time, trajectory, and stop were analyzed with Kruskal–Wallis tests; time data were log-transformed.The test choice followed normality checks: parking showed no significant deviation from normality, unlike time, trajectory, and stop.Source: B. Teleoperation Performance · p. 7
- p = 0.0461 for LiDAR versus Camera+LiDAR task completion time; no other pairwise time comparisons were significant.The reported effect size for this comparison was Cohen’s d = 0.830.Source: B. Teleoperation Performance · p. 7
VI. DISCUSSION
The discussion reports that Camera+LiDAR fusion aligns objective performance with subjective usability and workload measures, while LiDAR BEV offers stopping precision but weaker general navigation. The conclusion summarizes fusion’s advantages and identifies human-readable visualization and broader longitudinal testing as future work.
VI. DISCUSSION
- LiDAR-only and fused BEV offered greater stopping accuracy, but LiDAR-only had the highest collision rates and greatest trajectory deviation.The discussion attributes LiDAR’s stopping precision to its isometric perspective and its navigation weakness to a lack of semantic richness.Source: §VI · p. 7
- Fusion’s depth precision and semantic clarity trade-off is linked to co-located sensor placement, which participants said minimizes the transformation penalty during perspective shifts.The discussion states that the fused BEV maximizes safety margins while reducing operator mental workload.Source: §VI · p. 7
VII. CONCLUSION
- Camera+LiDAR yielded shorter task completion times than LiDAR-only BEV, with superior SUS and lower cognitive workload scores than single-sensor setups.The conclusion reports similar performance between fusion and Camera-only views, while Camera-only usability barely approached the perceived acceptable threshold.Source: §VI · p. 7, §VII · p. 8
- Future work will project camera textures onto 3D point clouds, use instance segmentation, and conduct longitudinal studies with a diverse participant pool.Source: §VII · p. 8