Source-linked AI summary
User Experience in Human-Machine Interaction: Insights from Field Studies in Autonomous Mobility
Helen Schneider, Svetlana Pavlitska, J. Marius Zöllner
TL;DR
Existing AV acceptance and UX research provides limited real-world, passenger-centered evidence combining affective self-assessment, user sensing, and machine data. This paper conducts eight field studies with 144 participants, develops integrated evaluation tools, and reports breathing, camera, and voice as feasible measurements while validating SAMotion against SAM. The work supports user-centered UX evaluation for autonomous mobility and robotics, within a scope focused on interaction during rides.
Problem
AV research lacks real-world passenger-focused studies and datasets that combine self-assessment, multi-modal user sensing, and autonomous-driving data.
Method
The paper conducts eight real-world AV studies with 144 participants using questionnaires, SAMotion real-time self-assessment, multi-modal user sensors, and machine data.
Results
Breathing, camera, and voice measurements were identified as feasible in real-world studies, while SAMotion strongly correlated with SAM for valence, arousal, and dominance.
Takeaways & Limitations
The resulting protocol, questionnaire, and app provide user-centered methods for evaluating autonomous-vehicle HMI and can be applied across robotics.
Takeaways & Limitations
The UX scope excludes vehicle design, boarding and exiting comfort, and waiting times, while EEG and heartbeat were dismissed because of data inconsistencies and discordant results.
Abstract
from arXiv · showhide
Autonomous vehicles (AVs) promise safer, cleaner, and more inclusive mobility, yet large-scale adoption is hindered by user acceptance rather than by technical challenges. Prior studies on acceptance and user experience largely rely on surveys, simulators or Wizard-of-Oz setups, often over-representing technologically enthusiastic participants and focusing on drivers instead of passengers. We address this gap with real-world field studies with AVs in real traffic, totaling 144 participants. Using multi-modal sensing, we evaluated EGG, heartbeat, breathing, camera and voice signals for affect inference in combination with vehicle data. Our results show that breathing, camera and voice measurements are reliable and pratical in naturalistic passenger contexts. We further contribute a validated study protocol, a self-assessment app for real-time assessment during human-machine interaction, and a tailored questionnaire to capture participant attitudes towards AVs. By grounding UX evaluation in real-world contexts, this work lays a foundation for user-centered design of autonomous mobility systems and robotics in general. Our work bridges the gap between affective computing and technical implementation of autonomous vehicles.
1 INTRODUCTION
Autonomous vehicles offer broad mobility benefits, but public acceptance remains constrained by psychological concerns and limited real-world passenger research. This work responds with field studies combining passenger affect, user sensors, and autonomous-driving data.
- AVs may reduce traffic accidents and CO2 emissions while improving mobility for underage, older, and impaired persons.
- Public acceptance of AVs remains limited, with a key obstacle identified as psychological rather than solely technical.
- Prior acceptance studies commonly use surveys, focus on drivers, and lack links to autonomous-driving field experiments.
- Eight real-world studies with 144 participants investigate passenger affect alongside autonomous-driving data using a multi-modal study protocol.
- The work contributes feasible breathing, camera, and voice measurements, a reusable study setup, an attitude questionnaire, and the SAMotion self-assessment app.
2 RELATED WORK
The related work frames UX as context-dependent and identifies a shortage of real-world, passenger-centered HMI datasets combining self-assessment, user sensing, and machine data. The paper therefore limits its UX focus to interaction during rides in moving autonomous vehicles.
- UX comprises perceptions and responses shaped by the user’s inner state, system characteristics, and interaction context.
- This work evaluates UX during direct HMI in a moving AV, excluding vehicle design features, boarding and exiting comfort, and waiting times.
- Affect models may be discrete, continuous, or hybrid, with continuous approaches commonly representing valence and arousal and sometimes dominance.
- Existing affect datasets commonly use camera and audio but generally omit robotic data, limiting combined evaluation of people’s feelings and HMI.
- Multi-modal passenger datasets combining self-assessment, user sensors, and machine data are scarce, while existing autonomous-driving datasets emphasize drivers or simulation.
3 METHODOLOGY
The methodology develops a field-study protocol that combines questionnaires, real-time affect self-assessment, and multi-modal sensing for autonomous-vehicle HMI. It iteratively uses study findings to refine the final setup and validate its measurement instruments.
- The research creates a real-world study protocol with sensor-collection code and derives the final setup from findings across successive studies.
- Questionnaires assess UX and AV acceptance using selected components of TAM, UTAUT, CTAM, and AVAM.
- The pre-questionnaire measures attitudes toward autonomous driving, AI, and new technologies before vehicle interaction.
- The post-questionnaire covers UX, perceived safety, trust, ease of use, transparency, control, enjoyment, overall rating, NPS, willingness to pay, and comments.
- Technological attitudes are aggregated from 1–5 Likert responses into Enthusiasts, Pragmatists, Conservatives, and Rejecters using predefined mean thresholds.
- Questionnaire validation uses Cronbach’s Alpha, Mann-Whitney U tests, and Cohen’s effect size for statistically significant constructs.
- SAMotion - App for Self-Assessment during Interaction: SAMotion enables real-time affect self-assessment during interaction by recording touch events and sending them to the machine for joint analysis with vehicle data.
4 EXPERIMENTS
Across eight studies with 144 participants, the authors iteratively evaluated autonomous-vehicle UX using questionnaires, self-assessment, vehicle data, and multiple user-sensing modalities. The experiments identified practical sensing options and showed that visualizations and real-time affect measures can capture passenger experience, while also exposing limitations in EEG, heartbeat, and voice analysis.
- Experimental program: Eight studies with 144 participants progressively evaluated autonomous-vehicle UX and refined the sensing and assessment setup.Each study used preceding results and insights to guide subsequent data collection and sensor selection.
- Questionnaire evaluation: An additional AV visualization significantly improved ease of use, enjoyment, fun, interest, and perceived sense of control among 36 passengers.The test group rated all three reported UX dimensions significantly better than the validation group.
- Self-assessment: SAMotion strongly correlated with SAM for valence (rs = 0.88; p = 0.00155), arousal (rs = 0.71, p = 0.0334), and dominance (rs = 0.9; p = 0.00103).The comparison involved 11 participants assessing videos with either SAM or SAMotion.
- Sensor evaluation: Raw EEG improved tablet-attention accuracy by 10%, and a GRU achieved 84% validation accuracy, but EEG was excluded later because of hardware inconsistencies and poor scalability.The EEG setup also faced sampling-rate variation, timestamp inconsistency, and challenges from more complex electrode configurations.
- Sensor evaluation: Heartbeat showed no significant correlations with machine data or affect, whereas breathing peaks aligned with route locations containing several interesting driving events.Some breathing hotspots lacked corresponding indications of interest, and missing camera or voice recordings limited interpretation of certain events.
- Final setup: HopeNet reached 97.31% accuracy for head-pose recognition after fine-tuning car and bike data, supporting camera use in the final study.The final setup combined camera, audio, tablet IMU, respiration-belt, machine, questionnaire, and SAMotion data.
5 CONCLUSION
The work develops user-centric methods for evaluating human experience in autonomous vehicles through real-world traffic studies, with relevance extending to robotics.
- The authors outline goals and challenges for real-world studies of autonomous vehicles using user-centric methods.
- These methods are presented as applicable beyond autonomous vehicles to the broader field of robotics.
- The work encourages incorporating humans into robotics development to improve acceptance and user experience in everyday life.