Source-linked AI summary
Human-Aware Target Tracking and Navigation: Fusing Kinematic State Estimation with Structural Map Constraints
Sagar Gupta, Don Gideon, Seng W. Loke, Kevin Lee, Bijo Sebastian
TL;DR
Visual trackers can lose targets or switch identities under occlusion, while short-horizon kinematic predictions can produce geometrically impossible paths after track loss. The system fuses visual detections, LiDAR clustering, and EKF tracking, then uses BFS on a topological map to predict feasible walking-lane trajectories when the track is lost. The map-informed recovery framework achieved a 71.4% reacquisition success rate, with time-to-reacquire taking up to 10 seconds per event.
Problem
Visual trackers can lose targets or switch identities under occlusion, while short-horizon kinematic predictions can produce geometrically impossible paths after track loss.
Source: §I · p. 1Method
The system fuses visual detections, LiDAR clustering, and EKF tracking, then uses BFS on a topological map to predict feasible walking-lane trajectories when the track is lost.
Source: §I · p. 1, C. Map-Informed Spatial Reasoning · p. 5Results
The map-informed recovery framework achieved a 71.4% reacquisition success rate, with time-to-reacquire taking up to 10 seconds per event.
Source: C. Experimental Scenarios · p. 6Takeaways & Limitations
Abstract
from arXiv · showhide
Autonomous mobile robots performing person-following tasks often suffer from temporary occlusions and sensor track loss in dynamic environments. This research presents an end-to-end autonomous navigation stack that addresses target occlusion through map-informed spatial reasoning. The proposed system features a multi-modal perception pipeline, fusing deep learning-based visual tracking with 2-dimensional LiDAR point clustering to maintain high-fidelity tracking of a tagged person. A continuous state estimator integrates this perception data with wheel odometry and IMU sensors for stable localization. When the active track is lost due to occlusion, the system activates a map-based recovery framework. Leveraging a predefined topological map, the system executes a graph-based search to propagate the target's last known trajectory along structurally defined walking lanes, adhering to left-hand regional conventions. By generating a discrete set of feasible future trajectories, the robot reasons about potential structural trajectory changes, such as continuing a heading or turning at an intersection. This map-informed prediction is fed directly to the local obstacle avoidance planner, enabling the robot to continue following its target safely and predictably until the person is visually reacquired. Real-world evaluations in dense multi-person environments demonstrate the system's robustness, achieving a 71.4\% target reacquisition success rate during major occlusion events lasting up to 7 seconds.
I. INTRODUCTION
Person-following robots must maintain a tagged target through temporary occlusions, where vision can lose identity and kinematic prediction alone can produce implausible paths. The paper combines multimodal tracking with map-based trajectory recovery and validates the stack on a differential-drive robot.
I. INTRODUCTION
- Vision-based tracking is vulnerable to occlusion and identity switches, while LiDAR alone cannot distinguish a specific tagged person from other people.When sensory tracking is lost, short-horizon kinematic prediction can yield geometrically impossible trajectories or a “freezing robot.”Source: §I · p. 1
- The system fuses visual tracking, 2D LiDAR clustering, and an EKF, then uses BFS over mapped walking lanes to predict target paths after track loss.The prediction is sent to the local DWA planner so pursuit can continue until visual reacquisition.Source: §I · pp. 1–2, Figure 2 · p. 2
- The work develops a hybrid tracking and occlusion-recovery framework and validates it on a differential-drive robot among up to 10 non-target pedestrians.The recovery strategy switches from kinematic EKF tracking to structural trajectory prediction using BFS.Source: §I · p. 2
II. RELATED WORK
Prior work combines visual identity tracking with spatial sensing and EKF estimation, while map-based motion hypotheses address the limits of kinematic prediction during extended occlusions. The paper situates its approach within these sensing, estimation, and planning lines of work.
II. RELATED WORK
- Vision-based identity tracking, vision-derived seeds with LiDAR clustering, and EKF fusion are established approaches for tracking targets in physical space.The related work also describes projecting camera boxes into LiDAR coordinates for clustering and localization.Source: §II · p. 2
- During temporary sensor interruptions, EKF systems commonly coast on the last motion model, with covariance tuning balancing sensor-jitter sensitivity against responsiveness to motion changes.The passage describes unicycle-based state formulations that track planar position and heading, with velocity inferred through temporal propagation.Source: §II · p. 2
- Topological maps can constrain motion hypotheses to feasible paths, including heading continuation or turns, and those predictions can inform uncertainty-aware local planning.The cited work also discusses left-side corridor conventions and proactive conflict resolution in shared spaces.Source: §II · p. 2
III. SYSTEM OVERVIEW
The system overview presents a three-tier architecture that links visual perception and spatial clustering with state estimation, map-informed recovery, and local navigation. Its dashboard depicts a tagged-person tracking setup using vision, LiDAR, and topological map information.
III. SYSTEM OVERVIEW
- The robot follows a tagged person using vision, LiDAR, and topological map information.The system overview identifies the tracked target with a green box and ID 55, while other people are shown with blue boxes.Source: Figure 3 · p. 3
- The architecture connects multimodal perception and continuous state estimation with map-informed recovery and a DWA planner for collision-free navigation.The localization stack fuses sensor data through EKF and AMCL, using an RGB-D camera, 2D LiDAR, IMU, and wheel encoders.Source: §III · pp. 2–3
- The operational dashboard integrates the system’s multimodal inputs during an active pursuit.The dashboard presents the robot’s camera, LiDAR, and map-based tracking context.Source: §III · p. 3
A. Vision
The vision subsystem detects people in RGB frames, maintains their identities across frames, and projects detections into the LiDAR frame. It uses a lightweight YOLOv8 nano detector for real-time pedestrian detection.
A. Vision
- The vision module extracts 2D bounding boxes from RGB frames and associates them across time before projecting them into the LiDAR frame.This establishes the perception module’s role in identifying and maintaining a target lock.Source: A. Vision · p. 3
1) Object Detection
- YOLOv8 nano generates pedestrian bounding boxes from the primary RGB stream for high-speed, real-time human detection.The lightweight model is selected to support low-latency updates for the robot’s low-level controllers.Source: 1) Object Detection · p. 3
- The object detector’s bounding boxes provide the input detections for the vision subsystem’s temporal association and LiDAR-frame projection.The detection pipeline is intended to minimize reaction time through low-latency updates.Source: 1) Object Detection · p. 3, A. Vision · p. 3
2) Object Tracking
Object tracking combines visual identity tracking with LiDAR-based spatial localization. Vision supplies a spatial seed for LiDAR clustering, while temporal tracking maintains a target estimate through temporary sensor loss.
2) Object Tracking
- DeepSORT maintains consistent person IDs across frames using spatial Kalman filtering and visual features, supporting tracking through temporary visual occlusions.The visual subsystem receives YOLOv8 bounding boxes and is managed by a centralized perception node.Source: 2) Object Tracking · p. 3
B. LiDAR-based tracking
- Fig. 4 depicts the vision feed and LiDAR scan, distinguishing traversable and obstructed velocities and marking the chosen velocity and tracked cluster.White dots denote point clouds; the orange circle marks the robot’s inflation radius.Source: Figure 4 · p. 3
- A homography projects the target’s 2D bounding box onto the ground plane as a seed for localized LiDAR clustering.The projection is used as a spatial seed because it is sensitive to camera pitch.Source: B. LiDAR-based tracking · p. 3
1) Clustering and Search
The system localizes a tagged person by clustering LiDAR points near a vision-derived seed, then uses an EKF to maintain a continuous motion estimate. During occlusion, the filter coasts on the last known velocity and can reset or prune a track under stated conditions.
1) Clustering and Search
- LiDAR points within 0.5 meters of the homography seed are clustered to isolate the target and produce a physical center-point coordinate.The scan is downsampled from a 3D point cloud to a 2D planar scan; the target-level EKF estimates (x, y, θ, v).Source: 1) Clustering and Search · pp. 3–4
- Fig. 5 illustrates map-informed future path branches diverging through an intersection according to walking conventions.The branches represent candidate target paths when line of sight is lost.Source: Figure 5 · p. 4
2) Extended Kalman Filter
- The EKF tracks each person with a four-dimensional state, X = [x, y, θ, v]T, where x and y are global planar coordinates, θ is heading, and v is linear velocity.It uses a prediction-and-update process; LiDAR observations provide x and y, while heading and velocity are inferred from position changes.Source: 2) Extended Kalman Filter · p. 4
- During occlusion, the EKF coasts on last-known velocity, resets to camera projection with inflated covariance if drift exceeds 1.0 meter, and prunes tracks after 5.0 seconds without LiDAR re-establishment.The filter assigns greater process uncertainty to heading and velocity than to spatial changes, reflecting possible abrupt human motion.Source: 2) Extended Kalman Filter · p. 4
C. Map-Informed Spatial Reasoning
When sensory tracking is lost, the system uses a topological graph and breadth-first search to propagate candidate target paths. The search explores structurally feasible routes and uses the occlusion context to prioritize branches.
C. Map-Informed Spatial Reasoning
- After sensory track loss, breadth-first search propagates the prediction along directed topological-map edges up to a predefined maximum path length.The graph represents structural nodes such as intersections and corners; the method addresses implausible short-horizon predictions such as paths through walls.Source: C. Map-Informed Spatial Reasoning · p. 4
- Fig. 7 shows map-based target-path predictions alongside the RGB target view and DWA velocity arcs, with traversable paths distinguished from collisions.The dashboard also marks the robot’s localized state, line-of-sight path, chosen velocity command, and safety inflation radius.Source: Figure 7 · p. 5
- At an intersection, the system explores diverging map paths and may deprioritize a straight path that remains in clear line of sight while propagating a turn that could explain occlusion.The candidate trajectories follow geometric constraints and the regional convention of walking on the left side of a corridor.Source: C. Map-Informed Spatial Reasoning · p. 5
A. Hardware Setup
The hardware validation uses a custom differential-drive robot with RGB-D, LiDAR, and IMU sensing. Evaluations take place in structured indoor corridors and intersections where left-side walking conventions inform map-based prediction.
A. Hardware Setup
- The validation platform is a differential-drive robot with a 0.65 m wheelbase and a conservative 0.45 m circular safety footprint.Its locomotion loop is capped at vmax = 0.4 m/s and ωmax = 1.5 rad/s.Source: A. Hardware Setup · p. 5
- The perception suite uses an Intel RealSense camera, a Velodyne VLP-16 LiDAR downsampled to a 2D scan, and an Xsens MTi IMU.Source: A. Hardware Setup · p. 5
B. Indoor Environment
- Evaluations use long orthogonal corridors and three-way intersections, where occupants primarily walk on the left side of hallways.These spatial constraints and conventions provide the basis for map-informed target prediction during occlusion.Source: B. Indoor Environment · p. 5
C. Experimental Scenarios
Across dense multi-person trials, map-based recovery reacquired the target in some major occlusions, while the evaluation also included a single-person erratic-movement scenario. The trials varied pedestrian density and occlusion duration.
C. Experimental Scenarios
- The evaluation covered five trials totaling 20 minutes; YOLOv8n averaged 16.2 FPS on 1280 × 720 RealSense feeds, and DeepSORT processed all detector bounding boxes.Source: C. Experimental Scenarios · p. 5
- In the foundational single-person scenario, the target made abrupt turns, sudden stops, and lateral side-to-side movements to test continuous EKF tracking and spatial reasoning.Source: C. Experimental Scenarios · p. 5
- The multi-person scenarios included 4, 6, 7, and 11 people, and Fig. 8 shows camera, LiDAR-cluster, and map views of example occlusions.The figure illustrates initial sensory-track loss and map-based reacquisition as the robot intercepts the predicted path.Source: C. Experimental Scenarios · pp. 5–6, Figure 8 · p. 6
- 71.4% reacquisition success was achieved across seven major occlusion events, with reacquisition taking up to 10 seconds; the scenes also produced five ID-switches and two re-tag failures.Visual blockage lasted 1–7 seconds in the multi-person scenarios.Source: C. Experimental Scenarios · pp. 5–6
V. CONCLUSION
The conclusion presents a person-following architecture that combines sensor tracking with map-informed recovery during occlusion. Real-world experiments validated tracking through erratic maneuvers and dense crowds, while dynamic crowd-flow prediction is left for future work.
V. CONCLUSION
- The architecture fuses visual inference, LiDAR point clustering, and continuous EKF state estimation, then uses a topological map to propagate target trajectories along expected walking lanes during sensory loss.Source: §V · p. 6
- Real-world experiments validated the robot’s ability to track targets through erratic maneuvers and dense crowds.Source: §V · p. 6
- Future work will integrate dynamic crowd-flow prediction to enhance adaptability in unstructured environments.Source: §V · p. 6