Source-linked AI summary
Gimbal-Based Human Tracking for Companion Robots Using Continual Learning
Cong-Thanh Vu, Ching-Chieh Liu, Yen-Chen Liu
TL;DR
Companion robots need continuous human tracking, but wearable tags reduce comfort and fixed cameras can lose users outside their limited fields of view. The paper combines active gimbal camera control with continual-learning ReID to maintain visual contact and adapt to appearance and environmental changes. Experiments report improved tracking stability and continuity, real-time re-identification, responsive tracking from walking to running, and better user experience without wearable tags.
Problem
Wearable tags reduce comfort, while conventional fixed-camera systems have limited fields of view that can prevent consistent observation as users move relative to the robot.
Method
The system combines a gimbal-mounted camera for active viewing-direction control with continual learning for adaptive person re-identification.
Results
The system significantly improves tracking stability and continuity, supports real-time re-identification, and tracks human motion from walking to running.
Takeaways & Limitations
User studies report improved user experience and interaction quality compared with tag-based tracking, while avoiding wearable hardware.
Abstract
from arXiv · showhide
Reliable and continuous human tracking is essential for natural human-robot interaction, particularly for companion robots. However, many existing approaches rely on wearable tags or fixed cameras with limited fields of view, which reduces system flexibility and often causes tracking failures when the target moves outside the sensing range. In this paper, we present a human tracking approach based on a gimbal-mounted camera integrated into a mobile robot. By actively controlling the gimbal mechanism, the camera can dynamically adjust its viewing direction to maintain the target within the field of view, even under substantial relative motion between the robot and the human. Furthermore, a continual learning strategy is applied to the person re-identification (ReID) task to adapt to changes in appearance and environmental conditions during long-term tracking. Experimental results demonstrate that the proposed system significantly improves the stability and continuity of human tracking, enables real-time re-identification, and provides responsive feedback for reliable tracking of human motion from walking to running. User studies further indicate that the proposed approach enhances user comfort by eliminating the need for wearable tags.
I. INTRODUCTION
Companion-robot tracking must keep users in view despite unpredictable motion, but tags impose comfort costs and fixed cameras have limited fields of view. This paper combines a gimbal-mounted camera with continual-learning ReID to maintain visual contact and improve tracking and user experience.
- Existing companion-robot systems use follow-behind, side-by-side, and front-following strategies, but adaptive tracking requires continuously keeping the user within the robot’s field of view.
- Tag-based tracking can localize users across relative positions, yet interference, failure conditions, and required wearable hardware reduce comfort.
- Conventional camera tracking remains constrained by limited field of view, preventing consistent observation when the user changes relative position alongside the robot.
- The proposed system uses a gimbal-mounted camera to maintain continuous visual contact when the user changes position.
- Continual learning adapts person re-identification to appearance, pose, and environmental changes, supporting more reliable long-term tracking.
- Experiments report superior recognition performance, improved user experience over tag-based tracking, and real-time tracking across movements from walking to running.
II. SYSTEM CONFIGURATION
This section describes the hardware configuration of the companion robot and its gimbal-based human-tracking camera system.
- The section covers the companion robot’s hardware configuration and its gimbal-based human-tracking camera system.
A. System Overview
The system combines an omnidirectional mobile base, environmental perception, centralized computation, and an independently actuated gimbal. Image-plane feedback keeps the user centered while decoupling camera orientation from mobile-base motion.
- The robot integrates an omnidirectional locomotion base, environmental perception suite, and centralized computing unit for real-time operation and navigation.
- The mobile platform independently controls translation and rotation, enabling flexible motion adaptation for human-accompanying behaviors.
- A 3D LiDAR supplies point clouds for a local costmap used in collision avoidance and safe motion planning.
- The Jetson Orin processing stack combines YOLOv8 detection, ByteTrack association, and ResNet-18 ReID with Online Continual Learning for adaptive identity tracking.
- An independently actuated gimbal decouples visual perception from mobile-base orientation and sudden maneuvers, maintaining stabilized visual contact during substantial relative motion.
- The ZED X camera uses a single-axis BLDC-motor gimbal with direct torque control to reduce cogging and vibrations during abrupt transitions from walking to running.
- A hierarchical ROS 2 architecture links the Arduino interface, high-level torque commands, PWM conversion, and SimpleFOC actuation.
- The gimbal controller uses pixel-coordinate error rather than 3D world coordinates because long-range depth errors can destabilize control and cause oscillatory behavior.
III. METHODOLOGY
The tracking pipeline combines detection, short-term association, ReID, and image-plane gimbal control. Pixel feedback and PID torque commands keep the recognized user centered despite limitations in depth estimation.
- YOLOv8 detects humans and ByteTrack associates detections across frames using Kalman filtering, including under low-confidence detections or partial occlusions.
- ResNet-18 ReID supplies feature vectors when motion cues are insufficient after prolonged disappearance, viewpoint changes, or scene re-entry.
- The recognized user is represented by image-plane centroid coordinates and a camera-frame real-world position, while depth estimates can degrade with distance.
- Pixel coordinates are used directly as controller feedback to mitigate depth-related vibrations and maximize gimbal tracking stability.
- The optical center p_target = [c_x, c_y]^T defines the desired image location, and the control error is the displacement between that center and the target position.
- A PID controller converts the image-plane error into required motor torque for continuous user tracking.
C. Human-Companion Controller
The human-companion task is formulated as an optimization problem solved by an MPPI controller. Its composite objective combines relative-position tracking, velocity synchronization, and obstacle avoidance, with behavior defined by the desired human-relative position.
- C. Human-Companion Controller: An MPPI controller computes control inputs from the robot’s current state and the human’s global-frame state.The human-companion task is formulated as an optimization problem.
- C. Human-Companion Controller: The composite cost promotes the desired relative position, synchronized human–robot velocities, and obstacle avoidance.The controller uses separate cost terms for tracking, velocity synchronization, and obstacle avoidance.
- C. Human-Companion Controller: Weighting factors w_t, w_v, and w_o respectively scale tracking, velocity synchronization, and obstacle-avoidance costs.The obstacle threshold r detects potential collisions using the local costmap, whose cost is obtained from LiDAR data.
- C. Human-Companion Controller: The desired relative position can place the robot in front of, behind, or beside the user.The desired configuration depends on the intended human–companion behavior and the human’s global position is estimated from camera-frame measurements.
IV. RESULTS AND DISCUSSION
The evaluation compares the proposed system with prior approaches across visual re-identification, user experience, and tracking under varied human behaviors. Experiments include slow walking and running conditions.
- IV. RESULTS AND DISCUSSION: The study compares the proposed method with prior works using the described companion-robot platform.The comparison covers object re-identification, user experience, and tracking performance.
- IV. RESULTS AND DISCUSSION: Object re-identification performance is evaluated in the vision system.
- IV. RESULTS AND DISCUSSION: Tracking performance is assessed during varied human behaviors, including slow walking and running.
A. Target Detection and ReID Evaluation
Tracking is evaluated under slow walking, brisk walking, and high-speed running. Errors broaden as motion intensity increases, but average errors remain near zero and the maximum Run-condition error stays below 0.4 m.
- A. Target Detection and ReID Evaluation: The system is evaluated under Slow, Fast, and Run target-motion conditions.These correspond to slow walking, brisk walking, and high-speed running.
- A. Target Detection and ReID Evaluation: Higher target speed increases tracking-error variance and widens the overall error range.The Slow condition has the narrowest error band, centered around a median of approximately –0.1 m.
- A. Target Detection and ReID Evaluation: Mean errors remain close to 0 m across motion conditions, while the maximum Run-condition error remains below 0.4 m.The results indicate responsive gimbal control during high-speed movement.
- A. Target Detection and ReID Evaluation: Unlike prior fixed-viewpoint and rear-only methods, the proposed system supports accompanying the user from front, side, and rear orientations.
1) Experimental Setup:
The experimental setup assesses tracking across human gait modes and user experience through participant ratings. Evaluation criteria cover autonomy, robustness, adaptability, re-identification, comfort, and companionship.
- 1) Experimental Setup:: A user study recruited ten participants aged 22–28 with no prior HRI experience.Each participant completed five testing scenarios and rated the experience on a five-point Likert scale.
- 1) Experimental Setup:: Tracking performance is examined across different human gait modes, with results presented in Fig. 4.
- 1) Experimental Setup:: User-study average scores are presented in Fig. 5.
- 1) Experimental Setup:: The study defines six qualitative metrics: Hands-free, Occlusion Robustness, Companionship, Target ReID, User Comfort, and Flexibility.
- 1) Experimental Setup:: Flexibility measures adaptation to walking patterns and maneuvering in dynamic environments.
2) Comparison Results:
The proposed method provides more balanced user-study performance than baseline systems, combining stronger occlusion robustness and comfort with the flexibility benefits of active tracking. Tag-based tracking remains strong for Target ReID but requires hardware and scores poorly on hands-free use and flexibility.
- Overall, the proposed method shows more balanced performance across all criteria than the baseline systems.
- The proposed method outperforms the fixed vision-based method in Occlusion Robustness, scoring 2.6 versus 2.5.The improvement is attributed to tracking algorithms that mitigate failure modes in cluttered environments.
- The proposed method maintains User Comfort at 4.3 and Companionship at 4.6, with a natural and non-intrusive perceived trajectory.
- The fixed vision-based approach scores 2.6 in Flexibility because its stationary camera constrains field of view and operational range.These restrictions reduce adaptation to rapid direction changes and complex trajectories.
- The tag-based system achieves Target ReID of 4.2 but scores 1.9 in Hands-free and 1.5 in Flexibility.Participants reported inconvenience from carrying additional hardware, which reduced natural movement and interaction quality.
C. Following Performance Evaluation
Outdoor experiments evaluated tracking across slow walking, normal walking, and running. Positional errors remained low during walking and bounded during running, while speed significantly increased error metrics without significant intersubject effects.
- The evaluation tested three volunteers on an 880-meter outdoor path at approximately 0.75, 1.2, and 2.45 m/s.These conditions represented Slow Walk, Normal Walk, and Run behaviors.
- Walking positional errors on the X and Y axes remained between 0.02 m and 0.04 m.The results showed high stability in both slow and normal walking conditions.
- During running, maximum average X and Y errors remained within approximately 0.11 m despite increasing error with speed.
- Movement speed significantly affected all three error metrics (p < 0.05), with errors increasing proportionally with velocity.
- Intersubject differences were not statistically significant for X-error, Y-error, or θ (p > 0.05).The study reports consistent tracking performance across participants’ gait characteristics and movement styles.
- The conclusion reports improved tracking stability and continuity across conventional fixed-camera and tag-based methods, with responsiveness from walking to running.