PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal-Language-Model-Driven Interaction and Companionship for Service Robots in Elderly-Care Facilities

Ching-Chieh Liu, Cong-Thanh Vu, Yen-Chen Liu

arXiv:2608.21387v1cs.ROeess.SY

TL;DR

Elderly-care robots need integrated companionship, natural interaction, and safety monitoring rather than isolated service functions. The paper presents a robot combining active visual following, LLM-driven speech interaction, and VLM-based safety assessment, achieving 93.3% task success versus 48.9% for the baseline and demonstrating effective interaction and fall-related monitoring.

  • Problem

    Existing elderly-care robot studies often focus on isolated service functions and lack integrated continuous companionship, natural interaction, and safety monitoring.

  • Method

    The system integrates active visual tracking, LLM-based dialogue and constrained tool execution, and VLM-based monitoring for fall-related and abnormal postural states.

  • Results

    93.3% (84/90) success rate, compared with 48.9% (44/90) for the baseline, under tasks requiring correct action selection, tool invocation, and physical execution.

  • Takeaways & Limitations

    The unified system supports human-following, context-aware interaction, semantic navigation, and emergency preemption in elderly-care scenarios.

Abstract

from arXiv · show

Service robots are increasingly deployed in elderly-care facilities to alleviate caregiver workload and enhance the quality of daily care. However, most existing studies focus on isolated service functions and lack integrated capabilities for continuous companionship, natural interaction, and safety monitoring. In this paper, we present an intelligent companion robot system that unifies active visual human-following, real-time LLM-driven speech interaction for intent understanding and task execution, and VLM-based safety monitoring for fall detection and abnormal posture assessment. The perception layer ensures robust human tracking and uses an active gimbal to maintain the user in view during occlusions or abrupt movements. At the interaction layer, a Large Language Model interprets spoken requests and maps them to robot actions, enabling escorting and semantic navigation. Simultaneously, a VLM-based safety agent continuously analyzes visual observations to detect fall-related or abnormal postures and triggers emergency responses when necessary. Experimental results demonstrate the system's ability to reliably follow and interact with humans, while effectively detecting potential falls to ensure user safety.

I. INTRODUCTION

The paper addresses the need for elderly-care robots that combine continuous human-following, natural dialogue, and integrated safety monitoring. It proposes a companion robot unifying active visual tracking, LLM-based interaction, and VLM-based fall detection.

  • Research gap: Elderly-care robots need natural dialogue and integrated safety monitoring because human-following alone cannot resolve indirect requests or support timely assistance.The paper identifies safety monitoring as a capability that should operate alongside interactive following rather than as an external add-on.
  • Proposed system: The proposed system integrates active visual tracking, LLM-based dialogue interaction, and VLM-based safety monitoring for elderly-care applications.These three components are presented as tightly coupled parts of the companion robot architecture.
  • Contributions: The visual tracking module uses a gimbal and tracking controller to keep the user in view and support stable, continuous human-following.The gimbal maintains camera coverage across movement or orientation changes, while the controller supports tracking stability.
  • Contributions: The LLM interaction layer interprets spoken requests and maps them to constrained robot functions and semantic navigation goals.This extends interaction beyond geometry-based distance regulation toward semantic navigation and caregiving assistance.
  • Contributions: The decoupled VLM safety agent assesses fall-related and abnormal postural states and can override ongoing behaviors to trigger emergency responses.Its role is to provide timely safety intervention in elderly-care facilities.

II. SYSTEM DESIGN

The hardware platform combines a holonomic mobile base, embedded computing, LiDAR, stereo vision, and an active gimbal. These components support navigation, perception, user tracking, and posture monitoring.

  • Hardware platform: The robot uses an AgileX SCOUT MINI mecanum-wheel platform for holonomic maneuverability.The platform dimensions are 612 × 580 × 245 mm.
  • Computing: An NVIDIA Jetson Orin module serves as the computing hub for real-time visual perception, SLAM localization, and communication with cloud-hosted LLMs.The module provides 2048 CUDA cores and 275 TOPS.
  • Perception sensors: Environmental perception fuses a Velodyne VLP-16 3D LiDAR with a ZED X stereo camera for localization, obstacle-aware navigation, and user monitoring.The stereo camera is connected through GMSL2.
  • Active monitoring: A single-motor active gimbal mounts the camera and supports user-centric monitoring, human tracking, and continuous posture analysis for fall detection.A PID-based controller adjusts the gimbal orientation to keep the user centered.

B. Software Architecture

The software architecture organizes the robot around ROS 2 components for perception, following, navigation, semantic interaction, and multimodal safety monitoring. Navigation and interaction operate with a parallel VLM safety agent.

  • Software architecture: The ROS 2 software architecture contains five components: skeleton detection and 3D localization, following control, mapping, semantic navigation, and multimodal LLM/VLM interaction with safety monitoring.The ZED SDK begins the perception pipeline by detecting humans and tracking skeletons from the gimbal-mounted camera stream.
  • Navigation: The Nav2 stack performs global and local path planning, obstacle avoidance, and motion execution from goal poses and velocity commands.It provides the navigation infrastructure for autonomous movement.
  • Interaction: The GPT-4o Realtime API supports speech interaction through bidirectional low-latency audio streaming and conversational-context maintenance.A dedicated Python ROS 2 node manages the audio exchange.
  • Safety monitoring: A parallel VLM safety agent continuously interprets visual observations and can preempt ongoing behaviors during safety-critical anomalies.The architecture gives the safety agent authority to trigger emergency escalation protocols.

III. METHODOLOGY

The methodology integrates visual following control, context-grounded LLM interaction, and VLM-based safety assessment. It uses active camera stabilization, coordinate transformations, semantic maps, and mode-specific control.

  • Integrated methodology: The methodology comprises visual perception and human-following control, an LLM interaction layer, and a VLM safety layer.These layers respectively support following, dialogue grounding and tool invocation, and fall-related state assessment with emergency escalation.
  • Visual perception and human-following control: The gimbal controller computes horizontal pixel error as eu = u − cx and uses PID control to drive the error toward zero.This maintains continuous visual contact during rapid target movements.
  • Visual perception and human-following control: Robot and human states are represented in global and local frames, with a coordinate transformation mapping relative human-state information from R to G.The rotation matrix J(θr) is identified as part of the transformation from the robot frame to the global frame.
  • LLM-driven dialogue interaction and tool invocation: The LLM converts spoken requests into constrained, safety-verified robot actions using operating mode, user utterances, and a semantic map as context.The semantic map links recognized zones to navigation goals, while the system prompt structures action generation.
  • Navigation and escorting: A mode-gated architecture assigns point-to-point navigation to Nav2 and active escorting to a relative-tracking controller.The controller adjusts relative position to maintain the desired tracking pose qd.

B. LLM-Driven Dialogue Interaction and Tool Invocation

The interaction layer grounds spoken requests in system context and a semantic map, then converts them into constrained, safety-verified robot actions. Runtime validation and environmental constraints prevent inappropriate tool execution.

  • Context-grounded decision process: The LLM combines user utterances, operating mode, and a verified semantic map to generate discrete high-level robot actions.The resulting action is selected from a validated action set rather than generated as unrestricted actuation.
  • Constrained tool invocation: Each executable action is deterministically mapped to ROS 2 tool-function calls within a strictly bounded action set.The bounded design supports predictable execution and prevents freeform actuation.
  • Constrained tool invocation: SLAM-map zone labels act as hard environmental constraints, so navigation commands are accepted only for recognized regions.Verified goals are sent to Nav2 for execution.
  • Runtime safety validation: Runtime checks block invalid or contextually inappropriate tool calls and redirect the interaction into clarification dialogue.The paper gives navigation during an emergency state as an example of a blocked action.

C. VLM-Based Safety Monitoring and Emergency Alert

A parallel VLM safety agent continuously analyzes RGB observations for falls and abnormal postures during interaction. Detected anomalies preempt robot behavior and trigger an alert pathway for timely intervention.

  • Continuous safety monitoring: The safety agent queries a VLM with a structured fall-assessment prompt using continuously captured RGB observations.It operates in parallel with the interaction agent to monitor fall-related incidents and abnormal postural states.
  • Emergency preemption: When an abnormal state is detected, the agent interrupts escorting or navigation and halts the robot’s motion.The binary emergency state is represented by F(t) ∈ {0, 1}.
  • Emergency alert: The anomaly is published through ROS to the LLM interaction layer, which issues A_alert to notify caregivers and escalate the situation.This decoupled design maintains safety monitoring independently of dialogue execution.

IV. RESULTS AND DISCUSSION

The evaluation covers human-following robustness, semantic interaction, user-perceived interaction quality, and VLM-based fall detection. Tracking experiments specifically examine performance across surfaces and walking speeds with and without gimbal stabilization.

  • Evaluation scope: The results section evaluates four aspects: human-following control, semantic interaction tasks, subjective interaction quality, and VLM-based fall detection.These experiments assess both system performance and interaction outcomes in an elderly-care context.
  • Tracking performance: Tracking experiments used a 100 m course with straight segments and curves across brick and asphalt surfaces, involving three participants at three walking speeds.The tested speeds were low (0.7 m/s), normal (1.2 m/s), and high (1.5 m/s).
  • Tracking performance: Tracking errors increase with target velocity, while active gimbal stabilization significantly mitigates deviations across speed profiles.Without the gimbal, lateral error surges at high speeds because base rotation is delayed during cornering.
  • Dynamic formation switching: An implicit request triggered a transition from back-following to side-by-side accompaniment, with longitudinal distance moving from 1.8 m to zero and lateral distance shifting to a 1.5 m offset.The maneuver was visualized through synchronized global trajectories and relative tracking profiles.

B. Evaluation of Semantic Interaction Tasks in Elderly Care

The LLM-based interaction module was evaluated on explicit, indirect, and multi-turn spoken instructions against a deterministic keyword-based baseline. It achieved higher end-to-end success, especially when requests were underspecified or required dialogue context.

  • Overall performance: Across 90 spoken trials, the LLM-based method achieved a 93.3% (84/90) success rate versus the baseline’s 48.9% (44/90).Success required correct action selection, valid tool invocation, physical execution, and multi-turn context handling.
  • Indirect requests: The LLM inferred intent from indirect requests such as “I’m thirsty,” whereas the rigid keyword-based baseline failed entirely.Both approaches performed well on explicit commands.
  • Multi-turn interactions: In multi-turn interactions, the LLM maintained dialogue context and updated task plans with 90% success versus the baseline’s 46.7%.The remaining LLM errors involved ambiguous phrasing that produced plausible but protocol-incompatible action mappings.

C. User Study

The user study compared visual-only, rule-based, and LLM-based following modes using five 5-point Likert dimensions. The proposed LLM-based mode received the highest ratings across all dimensions, particularly for natural interaction and intent understanding.

  • Ten participants aged 20–30 evaluated visual-only, rule-based, and proposed LLM-based human-following modes.Ratings used a 5-point Likert scale across five dimensions.
  • The LLM-based mode achieved the highest ratings across all five evaluation dimensions.The dimensions covered safety and trust, usability, naturalness, intent accuracy, and cognitive load.
  • It performed especially well in interaction naturalness and intent accuracy by interpreting indirect requests effectively.
  • By eliminating memorized commands, the LLM-based mode reduced cognitive demand and provided a more intuitive interaction experience.Higher cognitive-load scores represented lower burden.
  • The visual-only baseline scored lowest, while rule-based interaction improved moderately but remained constrained by rigid command templates.

D. Evaluation of VLM-Based Fall Detection

The system uses continuous VLM-based posture monitoring during active following to detect anomalous postures or falls. Detection preempts escorting, halts robot motion, and transitions the interaction layer into emergency assistance.

  • A VLM-based safety agent continuously monitors the user’s postural state from RGB observations during active following.
  • Upon detecting an anomalous posture or fall, the safety agent immediately preempts escorting and halts the robot’s motion.The halt is intended to mitigate secondary-injury risk.
  • After the physical halt, the LLM-driven interaction layer transitions the system into emergency assistance mode.
  • The complete system integrates active visual following, LLM-driven speech interaction, and VLM-based safety monitoring within a unified ROS 2 framework.Future work includes local edge deployment and evaluation in an actual elderly-care center.
Loading 2608.21387v1…