Source-linked AI summary
Establishing a Dynamic Multimodal HRI Dataset for Engagement Analysis with a Humanoid Robot
Buwan Kim, Wonse Jo
TL;DR
Prior HRI engagement research has relied primarily on observable behavioral cues, with physiological signals incorporated in some datasets. The paper proposes a structured collaborative-task experiment and synchronized dataset design integrating behavioral, physiological, human- and robot-motion, and post-task self-report measures. The proposed three-complexity-level protocol supports comparative analysis of interaction patterns and pairs objective sensor streams with subjective post-task annotations.
Problem
Prior HRI engagement research has relied primarily on observable behavioral cues, with physiological signals incorporated in some datasets.
Source: §1 · p. 1Method
The paper proposes a structured collaborative-task experiment and synchronized dataset design integrating behavioral, physiological, human- and robot-motion, and post-task self-report measures.
Source: §1 · p. 1, §2 · p. 1, §3.1 · p. 2Results
The proposed three-complexity-level protocol supports comparative analysis of interaction patterns and pairs objective sensor streams with subjective post-task annotations.
Source: §6 · p. 4Takeaways & Limitations
The data architecture is intended to provide a foundation for future multimodal analysis and investigations of interaction dynamics and user-state modeling in collaborative robotics.
Source: §6 · p. 4, §7 · p. 4The data will enable systematic investigation of multimodal interaction dynamics across varying task complexity levels.
Source: §7 · p. 4
Abstract
from arXiv · showhide
This paper presents an experimental design for constructing a multimodal dataset to analyze user engagement in human-robot interaction (HRI). Prior studies have mainly relied on observable behavioral cues, with limited frameworks integrating physiological signals. We therefore propose a structured data-collection protocol to build a multimodal dataset that includes wearable physiological signals, behavioral data, and self-report measures under different levels of task complexity defined in this experiment.
1. INTRODUCTION
Task-oriented human–robot collaboration makes multimodal interaction data important for studying engagement. The paper proposes an experiment and data-acquisition protocol combining behavioral, physiological, and synchronized motion measurements.
1. INTRODUCTION
- Task-oriented human–robot collaboration in domains such as manufacturing and logistics motivates collecting multimodal interaction data to study engagement.Source: §1 · p. 1
- Prior HRI engagement research has relied mainly on observable behaviors such as gaze, facial expression, and posture, with physiological signals included in some datasets.The cited examples include behavioral indicators and physiological measurements.Source: §1 · p. 1
- The proposed experiment integrates behavioral cues, physiological signals, and synchronized user- and robot-motion measurements in a multimodal HRI dataset design.It uses collaborative tasks such as object sorting and transport under varying task complexity.Source: §1 · p. 1
2. RELATED WORK
Prior HRI engagement studies have used behavioral cues and, in some cases, physiological data and subjective measures. This paper proposes a dataset design combining those modalities with synchronized human- and robot-side motion measurements.
2. RELATED WORK
- Prior work examined engagement through gaze, facial expressions, and posture, while other studies incorporated physiological signals and subjective self-reports.The cited behavioral examples include [Abrougui et al., 2017, Kompatsiari et al., 2018, Vanneste et al., 2021]; multimodal examples include [Anzalone et al., 2015, Oertel et al., 2014, Sorrentino et al., 2021].Source: §2 · p. 1
- The proposed dataset design combines behavioral indicators, EDA and PPG, and synchronized IMU measurements from both user and robot during collaborative tasks.The study builds on prior work on structured interaction in human–robot collaboration.Source: §2 · p. 1
- Table 1 is identified as a comparison of key features between the proposed and existing multimodal datasets.Source: Table 1 · p. 2
- Figure 1 depicts an experimental procedure with an introduction, main tasks across randomized zones, and a final data-integrity check.Source: Figure 1 · p. 2
3. PROPOSED EXPERIMENTAL PROTOCOL
The proposed protocol specifies a controlled, within-subject collaborative task experiment with three complexity conditions and a synchronized multimodal data architecture. It combines objective streams with post-task subjective assessments.
3. PROPOSED EXPERIMENTAL PROTOCOL
- The protocol organizes the session into an introduction, collaborative tasks, and a final data-integrity check, with stable baseline data and subsequent behavioral shifts as targets.Source: §3 · p. 2
3.1 Scenario: Multi-Stage Collaborative Tour Task
- Each participant completes three task-complexity conditions in randomized zone order, combining coordinated movement and object handling, with secondary tasks in some conditions.The protocol is planned for a controlled laboratory setting and follows informed consent with an introductory interaction.Source: §3.1 · p. 2
3.2 Participants and Inclusion Criteria
- The planned study recruits 30 volunteers, uses an a priori power analysis with f = 0.24, α = .05, and power = .80, and excludes participants with specified cardiovascular or neurological conditions or relevant medication use.Participants need normal or corrected vision and the physical ability to perform the collaborative tasks; prior robotics experience is not required.Source: §3.2 · p. 2
3.3 Experimental Design
- The three zones vary task demands: Zone A uses robot instructions and real-time corrective feedback, Zone B uses property descriptions and delayed feedback, and Zone C uses casual dialogue with clear targets and immediate confirmation.Source: §3.3 · p. 2
- After each zone, participants complete a 15-item, 5-point Likert questionnaire, with responses stored alongside the corresponding task condition.Source: §3.3 · pp. 2–3
- The design synchronizes physiological, wearable and robot-side IMU, joint-encoder, LiDAR, and RGB-D streams within a unified data architecture.The stated objective is synchronization of heterogeneous signals collected during human–robot collaboration.Source: §3.3 · pp. 2–3
4. MULTIMODAL DATA ARCHITECTURE
The data architecture pairs post-task subjective assessments with synchronized human- and robot-centric sensor streams. The questionnaire captures interaction-quality dimensions, while ROS2 provides temporal alignment for objective data.
4.1 Subjective Measures
- A 15-item post-task questionnaire rated on a 5-point Likert scale captures cognitive demand, communication quality, responsiveness, social perception, and satisfaction for each task condition.Responses serve as subjective annotations corresponding to each task zone.Source: §4.1 · p. 3
4.2 Objective Measures
- The human-centric stream records EDA, PPG, and wrist IMU, while the robot-centric stream includes joint encoders, robot IMU, RGB camera, and depth-based spatial measurements.Source: Human-Centric Stream · p. 3, Robot-Centric Stream · p. 3
- ROS2 temporally synchronizes objective streams within a unified logging framework for consistent multimodal alignment and real-time data collection.Source: Robot-Centric Stream · p. 3
5. DATA SYNCHRONIZATION AND PREPARATION
The protocol is designed to synchronize heterogeneous sensor streams and prepare them for multimodal analysis. Planned processing covers physiological, motion, spatial, and task-event data, with raw and preprocessed datasets retained alongside metadata.
5. DATA SYNCHRONIZATION AND PREPARATION
- All signals are planned to use a unified acquisition clock and be resampled onto a common timeline.This is intended to ensure consistency across heterogeneous data streams.Source: §5 · p. 3
- EDA and PPG are designed for band-limited filtering and outlier removal to suppress noise and mitigate motion artifacts.Both raw and filtered signals are intended to be retained for analytical flexibility.Source: §5 · p. 3
- Human- and robot-side IMU data are planned for gravity compensation, coordinate normalization, and unit standardization; encoder readings become angular-velocity profiles.Velocity-based thresholding is intended to extract robot motion-onset timestamps for segmentation.Source: §5 · p. 4
- RGB and depth-based spatial measurements are planned for temporal alignment and frame sampling, with relative distance and position stored as scalars.Source: §5 · p. 4
- Protocol timestamps are designed to segment task zones, while robot event logs mark events such as instruction onset and task completion.Raw and preprocessed datasets are planned to include sensor, coordinate, and preprocessing metadata for reproducibility and future usability.Source: §5 · p. 4
6. CONCLUSION
The proposed architecture combines multiple synchronized data streams for human–robot collaboration. It organizes collection across task-complexity levels and includes post-task assessments, providing a foundation for future multimodal analysis and user-state modeling.
6. CONCLUSION
- The proposed architecture integrates physiological signals, human motion, robot kinematics, and spatial context in a unified, time-synchronized framework.Source: §6 · p. 4
- Three task-complexity levels support structured comparison of interaction patterns, with subjective post-task assessments complementing sensor streams for each task zone.Source: §6 · p. 4
- The resulting data architecture is intended as a foundation for future multimodal analysis of interaction dynamics and user-state modeling in collaborative robotics.Source: §6 · p. 4
7. FUTURE WORK
Future work is to implement the proposed protocol and collect a large-scale dataset in the described collaborative scenarios. The dataset is intended to support systematic study of interaction across task-complexity levels and data-driven modeling of synchronized streams.
7. FUTURE WORK
- Future work will implement the protocol and collect a large-scale multimodal dataset under the described collaborative scenarios.Source: §7 · p. 4
- The acquired data will enable systematic investigation of multimodal interaction dynamics across varying task-complexity levels.Source: §7 · p. 4
- Subsequent research will explore data-driven modeling of synchronized physiological, kinematic, and spatial streams for multimodal interaction modeling and adaptive human–robot collaboration.Source: §7 · p. 4