Source-linked AI summary
When2Talk: When Should a Proactive In-Car Agent Talk?
Kaiser Hamid, Peihang Li, Nade Liang
TL;DR
Proactive in-cabin agents may help passengers understand automated-vehicle behavior, but speaking at every event can interrupt ongoing activity, and it is unclear when an event needs immediate speech, delayed speech, or none. In a controlled passenger-seat study, When2Talk compared event-triggered communication with a context-sensitive policy; context-sensitive communication increased appropriateness and substantially reduced perceived interruption, with outcomes varying by action.
Problem
It is unclear how to combine event and passenger-activity context when deciding whether a detected ride event warrants immediate speech, later communication, or no verbalization.
Source: §1 · p. 2Method
Results
Context-sensitive communication increased appropriateness and substantially reduced perceived interruption relative to event-triggered communication, with effects varying by action.
Source: §6 · p. 11Takeaways & Limitations
Event consequence, passenger activity, continuing information value, and confirmation need are considerations for selecting among Immediate, Delayed, and Silent communication.
Source: §7 · p. 13Delayed messages were delivered only at prespecified later opportunities, and further studies can test different delays, delivery opportunities, and measures such as recall and task-resumption time.
Source: §6.4 · p. 13
Abstract
from arXiv · showhide
Proactive in-cabin agents can help passengers understand automated-vehicle (AV) behavior, but communicating every ride event may introduce unnecessary interruptions. We investigated how communication should adapt to event priority and passenger activity. In a mixed-methods within-subject study, 41 participants rode as passenger in a VR simulated fully-automated vehicle. We compared an event-triggered (ET) policy that communicated immediately at every event with a context-sensitive (CS) policy that selected \textit{Immediate}, \textit{Delayed}, or \textit{Silent} communications. CS increased communication appropriateness and substantially reduced perceived interruption. Perceived trust did not differ between policies, although baselines dispositional trust differentiated communication preferences. Findings highlight event consequence, passenger activity, continuing information value, and confirmation need as key considerations for selective in-cabin communication.
1 Introduction
Passenger activity and ride context affect what passengers can see and when they can receive communication, while proactive messages can disrupt ongoing tasks. When2Talk compares immediate event-triggered communication with a context-sensitive policy that can speak immediately, later, or not at all.
1 Introduction
- Prior work links vehicle-action communication to passengers’ understanding and evaluations, while its value depends on situation and activity.Road-monitoring passengers may already see cues associated with a vehicle response that phone-focused passengers miss.Source: §1 · p. 2
- Poorly timed proactive messages can disrupt ongoing activities or reduce passengers’ sense of control.Disruption depends on task state, delivery time, and the effort of suspending and resuming activity; task boundaries can reduce it.Source: §1 · p. 2
- Less is known about whether detected ride events need speech at all, beyond the established importance of adapting timing to driving situation and passenger activity.Communication could be immediate, deferred, or left unspoken when vehicle motion and the scene already provide cues.Source: §1 · p. 2
- When2Talk compares a policy selecting Immediate, Delayed, or Silent communication with one that speaks immediately at every predefined ride event.Forty-one participants experienced both policies across six priority-by-activity contexts in an immersive CARLA passenger-seat study.Source: §1 · p. 2
- Context-sensitive communication increased appropriateness and substantially reduced perceived interruption while preserving communication value.The study also characterizes communication choices across passenger activities and ride events and provides a reusable CARLA-based passenger-seat VR testbed.Source: §1 · p. 2
2 Related Work
Automotive explanation research shows that communication evaluations depend on content, circumstances, and the recipient’s activity. Prior work on passenger attention and interruption motivates studying whether ride-event communication should be immediate, delayed, or omitted.
2.1 Explanations in Automated Vehicles
- Automotive explanation evaluations vary with content and situation, including perceived risk, timing, cultural context, correctness, and driving difficulty.Explanations of why a vehicle acts can improve understanding and trust relative to action-only descriptions, and speech can increase trust, usability, and anthropomorphic perceptions.Source: §2.1 · p. 2
2.2 Passenger Activity, Visual Access, and Attention
- Passenger activities change visual allocation and how much driving information occupants obtain directly, while spoken information can add cognitive demand.Occupants may self-interrupt non-driving tasks to inspect the road, and gaze varies with driving situation and automation mode.Source: §2.2 · p. 3
2.3 Timing and Selective In-Cabin Communication
- Interruption costs vary across task execution, and delivery near activity transitions can reduce perceived burden.Less is known about combining event and activity context to choose immediate, delayed, or no speech; When2Talk compares those policy choices while holding the canonical message constant when both policies speak.Source: §2.3 · p. 3
3 Research Questions
The study asks how event priority, passenger activity, and communication policy shape rider experience, how those factors interact, and how passengers reason about context-sensitive communication.
3 Research Questions
- The study examines how Event Priority, Passenger Activity, and Communication Policy affect appropriateness, interruption, trust, and usefulness.Source: §3 · p. 3
- It asks how Priority, Activity, and Policy interact in shaping passenger experience.Source: §3 · p. 3
- It asks how passengers perceive and reason about context-sensitive in-cabin communication.Source: §3 · p. 3
4 User Study
The study used a controlled within-subject design to compare event-triggered and context-sensitive communication across priority and passenger-activity contexts. A synchronized VR environment supported scripted rides, event-level ratings, and pupil-response measurement during roadway monitoring.
4 User Study
- Forty-one participants experienced both policies across six priority-by-activity contexts, with matched scenario variants preserving event meaning, vehicle response, message, and CS action.Source: §4 · p. 3, §4.8 · p. 7
4.1 Apparatus and Study Environment
- A CARLA passenger-seat environment used a Varjo headset and synchronized phone app to coordinate activities, scripted events, surveys, and eye-tracking records.The same canonical recording was used whenever both policies communicated for a given event.Source: §4.1 · pp. 3–4
4.2 Independent Variables and Experimental Contexts
- The repeated-measures factors were policy, event priority, and passenger activity, with Road Monitoring, Intermittent Phone Use, and Sustained Phone Use representing different attentional structures.Activity conditions differed in roadway access and availability of recurring task boundaries.Source: §4.2 · p. 4, §4.2.1 · p. 4
4.2.2 Ride Events and Communication Policies
- Each priority-by-activity context used a scripted ride event; CS actions were fixed before recruitment and selected as Immediate, Delayed, or Silent.Delayed messages were delivered at a prespecified opportunity 20 seconds later; the policy was not adaptive or learned during trials.Source: §4.2.2 · p. 4
4.3 Procedure
- Participants completed two policy blocks with three episodes each, producing six event-level observations per policy and 12 per participant; policy and scenario order were counterbalanced.Event questionnaires were completed between paired events, and participants answered open-ended questions after both blocks.Source: §4.3 · p. 4
4.4 Dependent Measures
- Event-level measures included appropriateness, interruption, trust in the communication decision, and usefulness, while block-level measures included appropriateness, disruption, satisfaction, and trust.Appropriateness was the primary event-level outcome; usefulness was analyzed separately because information value and delivery appropriateness can differ.Source: §4.4 · p. 4
4.5 Eye-Tracking and Pupil Response
- Pupil response during Road Monitoring was the change in mean pupil diameter from a one-second pre-event baseline to the scripted event period.Invalid eye-tracking samples were excluded before aggregation.Source: §4.5 · pp. 5–6
4.6 Data Analysis
- Repeated-measures tests and corrected paired comparisons assessed policy, priority, activity, and their interactions; open-ended responses were grouped into recurring communication rationales.Qualitative patterns concerned event consequence, passenger activity, continuing information value, interpretations of silence, and confirmation needs.Source: §4.6 · pp. 6–7
5 Results
Across the tested contexts, CS improved communication appropriateness and reduced perceived interruption relative to ET, while trust did not differ overall. Outcomes also varied by communication action, event priority, passenger activity, and baseline trust.
5.1.3 Trust (RQ1, RQ2).
- Trust did not differ overall by policy, but was higher in High-Priority than Low-Priority contexts (M=5.18 vs M=4.72).Source: §5.1 · p. 8, §5.1.3 · p. 8
5.1.4 Perceived Usefulness (RQ1, RQ2).
- ET usefulness was higher overall (M=4.51 vs CS M=3.71) and in all three Silent contexts; Delayed and Immediate contexts showed no significant difference.Source: §5.1 · p. 8, §5.1.4 · p. 8
5.1.1 Communication Appropriateness (RQ1, RQ2).
- Appropriateness favored CS in delayed blocked drop-off and ETA increase, and silent mild-slowdown contexts; the remaining three comparisons were nonsignificant after correction.Source: §5.1.1 · p. 8
5.1.2 Perceived Interruption (RQ1, RQ2).
- CS received lower interruption ratings in all five contexts where it delayed or withheld speech; ET was slightly lower in the Immediate emergency-vehicle context.Source: §5.1.2 · p. 8
5.1.5 Effects of Event Priority and Passenger Activity.
- Priority and activity interacted across outcomes; High–Low Priority differences were largest during Intermittent Phone Use for three outcomes and both phone-use activities for usefulness.Source: §5.1.5 · p. 8
5.1 Quantitative Results
- CS raised appropriateness (M=5.09 vs ET M=4.50) and lowered interruption (M=2.14 vs ET M=3.71), while trust was similar (M_ET=4.95; M_CS=4.96).Figure 6 presents participant-level policy scores averaged across priority and activity.Source: §5.1 · p. 8, Figure 6 · p. 9
5.1.6 Pupil Response During Road Monitoring.
- During both Road Monitoring events, pupil response was greater under CS, which withheld speech, than ET, which delivered the event message immediately.Source: §5.1.6 · p. 9
5.1.7 Individual Differences; 5.1.8 Block-Level Evaluation.
- Baseline trust tracked CS–ET differences in appropriateness, trust, and usefulness; block ratings showed lower disruption but no significant differences elsewhere.No association with interruption reached significance; age, gender, and prior AV experience did not differentiate event-level policy differences.Source: §5.1.7 · pp. 9–10, §5.1.8 · p. 10
5.2 Post-Study Interviews (RQ3)
Participants described five recurring considerations for choosing Immediate, Delayed, or Silent communication. Their preferences reflected event consequence, passenger activity, continuing information value, and the need for explicit confirmation.
5.2.1 Consequential events prompted a desire for immediate ac-
- Participants wanted immediate communication when events affected ride understanding, required preparation, or explained vehicle behavior, with concise acknowledgment sufficient for some.Source: §5.2.1 · p. 11
5.2.2 Passenger activity shaped timing preferences and perceived
- Passenger activity shaped timing preferences: participants wanted selective routine messages during engaging activities and fewer messages when roadway monitoring already exposed the event.Activity also shaped tolerance for small notifications and the information available from the surrounding scene.Source: §5.2.2 · p. 11
5.2.3 Deferral was acceptable when information remained useful.
- Delay was acceptable when information remained useful later; participants identified pauses and task boundaries as suitable delivery moments.The quantitative Delayed contexts showed higher CS appropriateness and lower interruption, without significant usefulness differences between policies.Source: §5.2.3 · p. 11
5.2.4 Silence reduced narration but removed explicit confirmation.
- Silence reduced narration when an event was visible or added little, but some participants valued acknowledgment as evidence that automation had detected it.Participants therefore saw silence as restraint that could also reduce available confirmation.Source: §5.2.4 · p. 11
5.2.5 Passengers differed in their need for explicit confirmation.
- Participants differed in their need for acknowledgment, message detail, and frequency; some preferred brief acknowledgment followed by detail adapted to urgency and activity.Responses linked timing preferences to event consequence, passenger activity, continuing information value, and confirmation needs.Source: §5.2.5 · p. 11
6 Discussion and Implications
Context-sensitive communication reduced interruption, but its value depended on whether information could be deferred or omitted and whether passengers wanted confirmation. The discussion identifies event consequence, passenger activity, information value, and confirmation need as design considerations, while noting limits in the tested policy mapping and delivery schedule.
6 Discussion and Implications
- CS increased communication appropriateness and substantially reduced perceived interruption relative to ET, with effects varying by communication action.In Delayed contexts, CS had higher appropriateness and lower interruption ratings, with no significant usefulness differences; in Silent contexts it reduced interruption while ET messages had higher usefulness ratings.Source: §6 · p. 11
6.1 Context-Sensitive Communication Generally Reduced Interruption
- CS reduced interruption in all five contexts where it delayed or withheld speech, and participants also rated the CS block as less disruptive.The discussion extends timing-focused work to whether an event should prompt immediate, later, or no speech.Source: §6.1 · pp. 11–12
- Perceived trust did not significantly differ by policy, while lower dispositional trust was associated with a greater preference for being kept informed.The discussion suggests prior trust may shape communication expectations alongside event and activity context.Source: §6.1 · p. 12
- Communication decisions should consider event consequence, passenger activity, and information already available, rather than suppressing speech based on attention alone.The design implication cautions against treating passenger attention as the sole basis for silence when safety or trip relevance is greater.Source: §6.1 · p. 12, §6.3 · p. 13
6.2 Delayed and Silent Communication Produced Different Trade-offs
- When information remains useful, deferral improved appropriateness and reduced interruption while usefulness remained comparable to ET.The proposed policy retains the message, seeks a more suitable delivery opportunity, and reassesses whether it remains relevant before speaking.Source: §6.2 · p. 12
- Silence reduced interruption in all three Silent contexts, but its appropriateness advantage appeared only for the mild slowdown; pedestrian and work-zone events may warrant explicit acknowledgment.ET messages retained a usefulness advantage in Silent contexts, indicating that reduced interruption does not mean the omitted information lacks value.Source: §6.2 · p. 12
6.3 Confirmation Needs Shaped the Value of Silence
- Participants valued spoken messages both for event information and for confirmation that the vehicle had detected or interpreted an event, especially for consequential events.A brief acknowledgment may suffice for confirmation, with explanation detail delayed or omitted depending on passenger knowledge and need.Source: §6.3 · pp. 12–13
6.4 Scope and Future Work
- The study tested one predefined CS mapping across six event-priority and passenger-activity combinations, with delayed messages delivered at prespecified later opportunities.Future work can compare alternative mappings, delay lengths, delivery opportunities, and additional measures such as recall, task-resumption time, and clarification requests.Source: §6.4 · p. 13
7 Conclusion
The study found that context-sensitive communication improved appropriateness and reduced interruption relative to event-triggered communication, with effects differing between delayed and silent delivery. It identifies event consequence, passenger activity, continuing information value, and confirmation need as considerations for adapting in-car communication.
7 Conclusion
- Compared with ET, CS increased communication appropriateness and substantially reduced perceived interruption, with effects varying by action.Delayed CS had higher appropriateness and lower interruption with no significant usefulness difference; in Silent contexts CS reduced interruption while ET messages had higher usefulness and provided valued confirmation for some participants.Source: §7 · p. 13
A Survey
The appendix describes the study questionnaires and response scale.
A Survey
- Questionnaire items generally used a seven-point agreement scale from 1 = Strongly disagree to 7 = Strongly agree.Source: A Survey · p. 16
- The documented materials include background questions, event-level and block-level questionnaires, and open-ended post-study questions.Source: A Survey · p. 16
- The appendix records the instruments used to collect participant responses across the study.It identifies the questionnaire types but does not enumerate their items in this passage.Source: A Survey · p. 16
A.1 Demographics and Prior Experience
Before the experimental rides, participants answered background questions about demographics, prior experience, baseline trust, and driving licensure.
A.1 Demographics and Prior Experience
- Participants reported age, gender, prior automated-vehicle experience, and prior virtual-reality experience before the rides.Source: §A.1 · p. 16
- Baseline AV trust was measured on a scale from 1 = Not at all to 7 = Completely.Source: §A.1 · p. 16
- Participants reported whether they held a valid U.S. driving license.Source: §A.1 · p. 16
A.2 Questionnaires
The questionnaires measured communication decisions after individual events and policy blocks, covering appropriateness, interruption or disruption, trust, usefulness, confidence, reliability, and satisfaction.
A.2.1 Event-Level Questionnaire
- Event-level ratings covered whether the communication decision was appropriate and whether it unnecessarily interrupted the passenger’s activity.Participants answered after each ride event, including when the vehicle spoke immediately, delayed a message, or remained silent.Source: §A.2.1 · p. 16
- Event-level ratings also asked whether passengers trusted the decision and found the communication useful for understanding..Source: §A.2.1 · p. 16
- All four event-level items used a seven-point agreement scale, with lower scores favorable only for perceived interruption.Source: §A.2.1 · p. 16
A.2.2 Block-Level Questionnaire
- After each three-episode policy block, participants rated overall timing appropriateness and disruption to their activity.Source: §A.2.2 · p. 16
- Block-level ratings also covered confidence, reliability, trustworthiness, and overall satisfaction with communication decisions.Source: §A.2.2 · p. 16
- The block questionnaire used six seven-point items, with lower scores favorable only for perceived disruption.Source: §A.2.2 · p. 16
- Confidence, reliability, and trustworthiness ratings were averaged into a block-level trust composite.Source: §A.2.2 · p. 16
A.3 Post-Study Questions
After both policy blocks, participants answered seven open-ended questions about when and how the vehicle should communicate.
A.3 Post-Study Questions
- The questions asked when the vehicle should speak immediately, wait, or remain silent.Source: §A.3 · p. 16
- Participants were asked how passenger activity and event importance should shape communication timing.Source: §A.3 · p. 16
- The questions also solicited disliked communication moments and rules participants would give the vehicle.Source: §A.3 · p. 16