Source-linked AI summary
Multimodal Takeover Requests for Drivers with Hearing Loss: Implications for AI-Enabled Communication in Automated Vehicles
Aries Chu, Wei-Hsiang Lo, Gaojian Huang
TL;DR
Drivers with hearing loss may not reliably access auditory takeover alerts, but evidence on visual and tactile SAE Level 3 communication remains limited. This simulator study tested signal type, information type, and hearing condition in 40 participants, finding that visual-tactile displays enabled faster responses while message content shaped response abruptness.
Problem
Evidence remains limited on visual and tactile takeover cues for drivers with reduced auditory access, especially in SAE Level 3 simulated-driving scenarios.
Method
A driving-simulator experiment with 40 participants tested three information types, three signal types, and normal-hearing versus simulated-hearing-impairment conditions.
Results
Visual-tactile displays produced the shortest reaction and takeover times, while baseline displays produced the shortest reaction times and informative content produced the lowest mean maximum resulting acceleration.
Takeaways & Limitations
Advisor AI can jointly adapt takeover signal modality and message content to the available time and the desired maneuver quality.
Takeaways & Limitations
Hearing impairment was simulated with earplugs and noise-canceling headphones in participants without reported hearing loss, which cannot reproduce long-term perceptual adaptations.
Abstract
from arXiv · showhide
More than 430 million people worldwide live with disabling hearing loss. Although people with hearing loss are legally permitted to drive and may benefit from conditionally automated vehicles, SAE Level 3 systems still require drivers to respond to takeover requests when automation reaches its limits. Existing takeover requests often rely on auditory information, yet little evidence addresses visual and tactile designs for drivers who cannot rely on sound. This driving-simulator study with 40 participants examined the effects of information type (instructional, informative, and baseline), signal type (visual, tactile, and visual-tactile), and hearing condition (normal hearing and simulated hearing impairment) on takeover performance. Information type significantly affected reaction time, with baseline displays producing the shortest times. Signal type significantly affected reaction and takeover time, with visual-tactile displays producing the shortest times. The interaction between signal type and information type was significant for all three measures. Visual-tactile displays produced the shortest reaction times within every information type. With visual-tactile signaling, simple baseline alerts prompted the fastest reactions and the most abrupt maneuvers, whereas informative content produced the lowest mean maximum resulting acceleration. Hearing condition showed no significant main effect on any measure. These findings suggest that AI-enabled vehicles can support urgent takeover communication through visual-tactile displays and can adapt message content to the time available and the maneuver quality required, with implications for drivers across hearing abilities.
I. INTRODUCTION
Hearing loss creates a need for takeover communication that does not depend on auditory access. This study compares visual, tactile, and combined signals, information content, and hearing condition in SAE Level 3 takeover requests.
- More than 430 million people worldwide experience disabling hearing loss, highlighting the need to consider hearing access in vehicle safety design.
- SAE Level 3 automation requires drivers to resume manual control when the system reaches operational limits or fails, often within limited time.
- Prior multimodal-warning research often includes auditory channels, leaving limited guidance for drivers who cannot reliably access sound.
- Within the AI collaboration framework, Advisor AI is the relevant role during a takeover request because the driver has not yet resumed manual control.
- Evidence comparing instructional and informative takeover content is mixed, and few studies test whether information effects depend on signal type.
- The study examines signal type, information type, and hearing condition to clarify visual-tactile takeover communication for drivers with reduced auditory access.
A. Participants
The study analyzed 40 participants using a medium-fidelity driving simulator, with normal-hearing and simulated-hearing-impairment groups experiencing visual and tactile takeover stimuli.
- Participants: 42 participants were recruited, and data from the remaining 40 participants, divided equally between hearing conditions, were analyzed.
- Participants: Participants were recruited from San Jose State University’s research pool, were 18–44 years old, and drove an average of 11 hours per week.
- Apparatus: The experiment used miniSim, a medium-fidelity simulator with three forward LED screens, an odometer display, steering controls, pedals, and a driver’s seat.
- Stimuli: Visual stimuli used a red circle, green arrow, or approaching-car depiction, while tactile signals were delivered through seat-mounted tactors vibrating at 250 Hz.
- Hearing condition: Simulated hearing impairment was produced with earplugs beneath noise-canceling headphones playing white noise, reducing ambient auditory input.
C. Experimental Design
The experiment used a mixed factorial design crossing three information types, three signal types, and two hearing conditions across repeated takeover events.
- Design: The study employed a 3 × 3 × 2 mixed factorial design involving information type, signal type, and hearing condition.
- Design: Information type and signal type were within-subject factors, whereas hearing condition was a between-subject factor.
- Procedure: Each participant completed three drives, one per information type, with six takeover events per drive and randomized signal-type order.
- Procedure: Training familiarized participants with the simulator, visual and tactile cues, takeover events, and expected actions before experimental drives.
- Task: During takeover events, participants first braked to deactivate automation, then steered around construction zones and later returned to the middle lane.
E. Dependent Measures
Takeover performance was evaluated through reaction time, takeover time, and maximum resulting acceleration, analyzed with factorial ANOVAs and corrected pairwise comparisons.
- Measures: Reaction time measured the interval from signal onset to the driver’s first steering-wheel movement or brake-pedal press.
- Measures: Takeover time measured the interval from signal onset to a 2° steering-wheel change indicating initiation of evasive steering.
- Measures: Maximum resulting acceleration was the largest vector magnitude of longitudinal and lateral acceleration during the post-takeover phase.
- Interpretation: Higher maximum resulting acceleration indicated a more abrupt transition, whereas lower values indicated a smoother takeover.
- Analysis: Separate 3 × 3 × 2 mixed ANOVAs tested information type, signal type, and hearing condition for each dependent measure.
III. RESULTS
Reaction time varied by signal and information type. Visual-tactile displays were fastest across information types, while baseline content generally produced the shortest reactions.
- Signal type: VT displays produced shorter reaction times than V and T displays, which did not differ significantly.VT: M = 1.378 s; V: M = 1.612 s; T: M = 1.648 s.
- Information type: Baseline displays produced shorter reaction times than instructional and informative displays.Baseline: M = 1.451 s; instructional: M = 1.607 s; informative: M = 1.580 s.
- Interaction: The interaction between signal type and information type was significant.F(2.644, 100.481) = 3.728, p = .017.
- Interaction: VT displays produced shorter reaction times than V and T displays within every information type.All pairwise comparisons were significant at p ≤ .004.
- Information-by-signal effects: Within T and VT displays, baseline content produced shorter reactions than selected richer-content conditions.For T, baseline was faster than instructional and informative content; for VT, baseline was faster than informative content.
- Hearing condition: Neither interaction involving hearing condition was significant.Signal type × hearing condition: p = .344; information type × hearing condition: p = .130.
B. Takeover Time
Takeover time was shortest with visual-tactile displays, and its dependence on information type varied by signal type. Hearing condition and information type showed no significant main effects.
- Signal type: Signal type significantly affected takeover time.F(1.738, 66.038) = 18.874, p < .001.
- Signal type: VT displays produced shorter takeover times than V and T displays, which did not differ significantly.VT: M = 2.125 s; V: M = 2.502 s; T: M = 2.554 s.
- Main effects: Information type and hearing condition did not significantly affect takeover time.Information type: p = .092; hearing condition: p = .473.
- Interaction: The interaction between signal type and information type was significant.F(3.253, 123.626) = 8.875, p < .001.
- Interaction: With instructional content, T displays produced longer takeover times than V and VT displays.T instructional takeover time: M = 2.940 s; comparisons with V and VT were significant.
- Hearing condition: Neither interaction involving hearing condition was significant.Signal type × hearing condition: p = .845; information type × hearing condition: p = .479.
C. Maximum Resulting Acceleration
Maximum resulting acceleration depended on the combination of signal and information type rather than on any single factor alone. Visual-tactile signaling had the lowest acceleration with informative content but the highest with baseline content.
- Main effects: No significant main effects of signal type, information type, or hearing condition were found for maximum resulting acceleration.The corresponding p values were .082, .082, and .323.
- Interaction: The interaction between signal type and information type was significant.F(3.714, 141.127) = 10.550, p < .001, η2 p = .217.
- Signal-by-information effects: With baseline content, VT produced higher maximum resulting acceleration than V and T.VT: M = 9.077 m/s2; V: M = 8.152 m/s2; T: M = 8.107 m/s2.
- Signal-by-information effects: With instructional content, V produced higher acceleration than VT, while with informative content, T exceeded V and VT.These pairwise differences were significant.
- Visual-tactile displays: Within VT displays, baseline content produced the highest acceleration and informative content the lowest.Instructional: M = 8.122 m/s2; informative: M = 7.601 m/s2.
- Hearing condition: Neither interaction involving hearing condition was significant.Signal type × hearing condition: p = .354; information type × hearing condition: p = .425.
IV. DISCUSSION
Baseline content yielded the shortest reaction times, whereas richer content did not show the same main-effect advantage on takeover time or acceleration. Its effects depended on signal type.
- Information type: Baseline displays produced the shortest reaction times, while instructional and informative displays did not differ reliably.The authors suggest baseline familiarity may have contributed to this reaction-time advantage.
- Outcome-specific effects: The baseline reaction-time advantage did not appear as a main effect on takeover time or acceleration.Interaction results nevertheless showed that baseline consequences depended on signal type.
B. Effects of Visual-Tactile Signaling
Visual-tactile signaling produced the fastest takeover responses, while its benefits depended on information content and extended to maneuver quality. Hearing condition did not significantly affect reported takeover measures.
- Signal type: Visual-tactile displays produced shorter reaction and takeover times than visual or tactile displays alone.Multiple-resource theory is offered as a possible explanation because visual and tactile information can be processed in parallel with less interference.
- Interaction effects: The signal-type and information-type interaction qualified the main effects across reaction time, takeover time, and resulting acceleration.Baseline content was fastest with tactile and visual-tactile displays, while visual-tactile signaling produced the shortest reaction times within each information type.
- Interaction effects: Baseline content prioritized rapid detection, whereas informative content supported smoother control when additional processing time was available.This distinction maps onto the signal-response and post-takeover phases of the two-phase takeover framework.
- Hearing condition: Hearing condition produced no significant main effects on the reported takeover measures.All requests used visual, tactile, or combined signaling, so the auditory channel conveyed no task-relevant information in this experiment.
E. Implications for AI-Enabled Takeover Communication
The findings support joint adaptation of signal modality and message content in AI-enabled takeover communication. Visual-tactile alerts can favor speed, while content and timing should also account for maneuver smoothness and the study’s design limits.
- AI-enabled communication: An Advisor AI should treat signal modality and message content as a joint design decision for urgent takeover communication.Simple alerts suit short time budgets, whereas informative messages suit longer lead times that allow maneuver preparation.
- AI-enabled communication: Simple multimodal alerts traded faster reactions for more abrupt control, whereas informative content supported smoother takeover maneuvers.Instructional tactile messages warrant caution because the tactile-only instructional condition produced the longest takeover times.
- AI-enabled communication: Visual-tactile signaling consistently supported faster responses, while informative content produced lower mean maximum resulting acceleration.With a short time window, the authors suggest a simple VT alert; with more lead time, an informative VT message may support maneuver preparation.
- Limitations and future work: The study used simulated hearing impairment and a young university-affiliated sample, limiting direct generalization to drivers with actual hearing loss and other age groups.Tactile perception also varied across participants, motivating calibration and alternative cue locations in future designs.