Source-linked AI summary
Development of a Humanoid Robot Prototype for Multimodal Human-Robot Interaction
Thang Tran Viet, Thanh Nguyen Canh, Huy Uong Gia, Phuc Dinh Van, Son Tran Duc, Ngoc Minh Do, Xiem HoangVan
TL;DR
Existing humanoid platforms trade off accessibility against capability for flexible HRI research. This paper presents an affordable, modular prototype integrating multimodal AI, achieving high recognition accuracy and approximately 90% pick-and-place success with 1.83 cm average manipulation error.
Problem
Existing humanoid platforms remain either high-performance but inaccessible or affordable but underpowered for flexible HRI research and experimentation.
Method
The authors develop a fully 3D-printed humanoid platform with multimodal AI modules for gesture recognition, object detection with stereo-based 3D localization, and speech-driven LLM semantic parsing.
Results
The prototype achieved 96% gesture-recognition accuracy, 92% speech-transcription accuracy, over 90% object-detection accuracy, approximately 90% pick-and-place success, and 1.83 cm average manipulation error.
Takeaways & Limitations
The results support the prototype as an accessible, modular platform for real-world HRI research and prototyping.
Abstract
from arXiv · showhide
Human-robot interaction (HRI) enables intuitive and intelligent collaboration between humans and robots in real-world environments. This paper introduces a humanoid robot prototype designed as a flexible testbed for developing and integrating artificial intelligence (AI) modules in HRI tasks. The system features a 12 degree-of-freedom (DOFs) dual-arm mechanism and a 2 DOFs head with an expressive LCD screen to express facial emotions. All hardware components are controlled by a custom-designed controller board with real-time AI processing supported by an onboard Jetson module. The system incorporates three AI modules: (1) gesture recognition using MediaPipe Pose and an LSTM classifier, (2) object detection with YOLO and 3D localization, and (3) voice-command processing through speech recognition and large language model(LLM)-based semantic parsing. The platform is validated through experiments on positioning accuracy, with results showing average manipulation errors of approximately 1.83 cm. To demonstrate its versatility, experimental results show over 90% task accuracy, with gesture recognition reaching 96%, speech recognition reaching 92%. The results confirm the effectiveness of the proposed system as a reproducible and accessible humanoid platform for research and prototyping in HRI.
I. INTRODUCTION
Existing humanoid platforms either provide high performance at limited accessibility or remain affordable but underpowered for versatile HRI. This work presents an operational, low-cost prototype integrating modular hardware with real-time AI across speech, vision, and gesture interaction.
- Research gap: Open-source educational humanoids improve accessibility but remain limited in strength, embodiment, or versatility for complex HRI.NimbRo-OP2X, ROBOTIS OP2/OP3, and PoppyHumanoid illustrate these trade-offs.
- Proposed platform: The proposed prototype targets the gap between high-performance yet inaccessible systems and low-cost but underpowered alternatives.It is designed for academic research and HRI experimentation in real-world human-facing scenarios.
- Contributions: The platform combines a 12-DOF dual-arm mechanism, 2-DOF head, 7 in LCD, ROS, and onboard GPU-enabled Jetson processing.The LCD supports expressive facial interaction, while the Jetson supports AI inference.
- Contributions: Three AI modules support gesture recognition, stereo-based 3D object localization, and natural-language understanding through speech recognition and LLM-based semantic parsing.The system integrates these modalities within one humanoid platform.
- Evaluation: Two real-world HRI scenarios evaluate gesture-based interaction and voice-command-driven object manipulation quantitatively.The evaluation examines accuracy, latency, and task execution.
II. METHODOLOGY
The prototype uses a modular mechanical–electronic design combining aluminum structure, 3D-printed components, dual-arm actuation, expressive sensing, and layered computing. A custom controller board connects low-level motor control with Jetson-based AI processing.
- Mechanical structure: The robot uses lightweight aluminum bars with custom 3D-printed joints and brackets for a modular mechanical structure.The hybrid construction supports load-bearing components alongside rapid fabrication and replacement.
- Mechanical structure: At approximately 150 cm and 15 kg, the human-scale prototype supports natural-height gestures and object handovers.The stated examples include handshakes, waves, and object handovers.
- Actuation: The dual-arm system has 12 DOFs, with six revolute joints per arm supporting manipulation, expressive gestures, and physical interaction.Arm kinematics follow the Denavit–Hartenberg convention for forward and inverse computation.
- Sensing and expression: A 2-DOF head, 7 in LCD, and ZED 2 stereo camera provide expressive attention, facial animation, and RGB-depth perception.The custom controller board coordinates the electronic components.
- Computing architecture: The Jetson AGX Xavier runs ROS and AI workloads while microcontrollers handle low-level motor control through a custom controller board.This layered architecture separates real-time control from high-level perception, planning, and decision-making.
- Power and communication: A 36V supply is stepped down to 19V for Jetson processing and 5V for controller modules and servos.The ZED 2 camera and microphone connect directly to the Jetson, while servo signals are managed through the controller board.
B. AI Integration and Software Architecture
The software architecture comprises three modular AI modules that collectively enable the robot to perceive, understand, and respond to human interaction cues.
- B. AI Integration and Software Architecture: Three modular AI modules organize the software system: object detection, gesture recognition, and voice-command interpretation.Together, they cover perception, interaction understanding, and response generation.
1) Object recognition:
The object-recognition module combines YOLO detection on RGB images with stereo depth to localize detected objects in 3D coordinates.
- Object recognition: YOLO identifies object classes and pixel-space bounding boxes from the ZED2 RGB stream.The detector operates on RGB data acquired alongside depth information.
- Object recognition: Detected pixel coordinates are projected into 3D world coordinates using the ZED2 stereo camera’s depth map.Stereo-camera intrinsic calibration with a chessboard pattern is used to improve localization accuracy.
2) Gesture recognition:
The gesture-recognition module uses a three-stage pipeline to classify predefined gestures from real-time human pose sequences. It combines MediaPipe Pose, temporal processing, and an LSTM trained on varied gesture conditions.
- The module comprises human pose estimation, temporal feature extraction, and gesture classification.
- MediaPipe Pose extracts real-time 2D keypoints, while an LSTM classifies sequences into waving, handshaking, and love gestures.
- The LSTM was trained on custom gesture sequences recorded under varied lighting and distance conditions.
3) Voice Command Understanding:
The voice-command pipeline converts spoken instructions into structured robot actions by combining speech recognition, LLM-based semantic parsing, visual localization, and motor execution.
- Speech input is transcribed and passed to an LLM that parses commands into structured action-object-location information.
- Detected objects and locations provide 3D coordinates that are used to translate parsed instructions into robot commands.
- Commands such as picking and moving are forwarded to the motor control unit for task execution.
- The voice-command scenario was evaluated as one of two real-world HRI experiments in an indoor laboratory environment.
A. Experimental setup
The experimental setup places the humanoid robot on a fixed base with head-mounted stereo vision and nearby voice input. Repeated trials assess response accuracy, latency, and robustness while gesture interactions trigger physical and expressive responses.
- The robot stood on a fixed base while a ZED2 stereo camera mounted on its head provided visual perception.
- A directional microphone positioned near the user captured voice input during the experiments.
- Each experiment was repeated multiple times to evaluate response accuracy, latency, and robustness.
- At approximately 1.5 m, recognized gestures triggered predefined arm motions and emotional expressions on the LCD screen.
2) Scenario 2: Voice-command Execution and Object Detection:
The voice-command scenario combines spoken instruction parsing with YOLO-based 3D object localization and inverse-kinematics control. The complete AI system was evaluated both independently and in combination for accurate, natural HRI responses.
- Verbal commands are transcribed, parsed by an LLM into executable instructions, and paired with ZED2-based object localization.
- YOLO detects and localizes objects in 3D space, while inverse kinematics converts the resulting sequence into pick-and-place operations.
- The AI modules were evaluated independently and in combination to assess accurate and natural responses in HRI scenarios.
1) Gesture Recognition Performance:
The prototype combines gesture, voice, and vision modules for interactive manipulation, achieving high recognition accuracy and approximately 90% pick-and-place success. Positioning errors remain low but increase when YOLO detection and coordinate transformation are included.
- Gesture Recognition Performance:: 96% gesture recognition accuracy was achieved across three predefined gestures, with a 1.2-second average response latency.The robot produced responses such as waving, smiling, audio playback, or handshakes.
- Voice Command and Object Detection Accuracy:: 92% speech transcription accuracy and 96% semantic parsing accuracy were achieved on 300 randomly issued spoken instructions.Structured JSON outputs helped generate actions in about 2 seconds on average.
- Voice Command and Object Detection Accuracy:: Over 90% object-detection accuracy was maintained across varying lighting conditions and object types.The YOLO-based pipeline enabled successful pick-and-place execution after command parsing and object detection.
- Positioning and Manipulation Precision:: Approximately 1.83 cm average positioning error and approximately 90% pick-and-place success were achieved after perception and control integration.YOLO detection and transformation noise introduced additional error.
IV. CONCLUSION
The paper presents and evaluates a modular humanoid robot for real-world HRI, integrating expressive hardware with gesture, object-detection, and language-understanding modules. The system achieved up to 96% gesture accuracy, approximately 90% voice-command execution success, and less than 1.9 cm average pick-and-place positioning error, while relying on external LLM APIs.
- IV. CONCLUSION: The prototype integrates a 12-DOF dual-arm mechanism, a 2-DOF expressive LCD-equipped head, and three AI modules for gesture, vision, and language interaction.The modules use MediaPipe Pose with LSTM, YOLO with stereo localization, and speech recognition with LLM semantic parsing.
- IV. CONCLUSION: 96% gesture accuracy, approximately 90% voice-command execution success, and less than 1.9 cm average pick-and-place positioning error were reported.These results cover the paper’s tested interaction and manipulation scenarios.
- IV. CONCLUSION: External LLM APIs introduce latency and dependence on internet connectivity, limiting the system’s independence from network services.This is identified as a key limitation and motivates future work on the platform.