PaperlayerView exact revision on arXiv

Source-linked AI summary

Person Following by Autonomous Robots: A Categorical Overview

Md Jahidul Islam, Jungseok Hong, Junaed Sattar

arXiv:1803.08202v4cs.RO

TL;DR

Autonomous person-following is needed across diverse human-robot applications, but operating environments and application requirements constrain sensing, autonomy, and robot dynamics. This paper categorizes the literature, reviews methods and operational challenges across ground, underwater, and aerial scenarios, and qualitatively compares approaches while identifying open problems.

  • Problem

    Person-following spans diverse applications and environments whose sensor, autonomy, and dynamic constraints make a generic methodology difficult to design.

  • Method

    The paper categorizes person-following behaviors, reviews design issues and state-of-the-art perception, planning, control, and interaction methods, and qualitatively analyzes their feasibility.

  • Results

    The review provides a comprehensive overview, compares prominent approaches across operational considerations, and presents their applicability in varied scenarios.

  • Takeaways & Limitations

    The categorization and comparison organize application-specific design choices and identify open problems for future person-following research.

  • Takeaways & Limitations

    Deep learning methods require comprehensive training samples and can perform poorly under occlusion, target-appearance changes, and environmental changes.

Abstract

from arXiv · show

A wide range of human-robot collaborative applications in diverse domains such as manufacturing, health care, the entertainment industry, and social interactions, require an autonomous robot to follow its human companion. Different working environments and applications pose diverse challenges by adding constraints on the choice of sensors, the degree of autonomy, and dynamics of a person-following robot. Researchers have addressed these challenges in many ways and contributed to the development of a large body of literature. This paper provides a comprehensive overview of the literature by categorizing different aspects of person-following by autonomous robots. Also, the corresponding operational challenges are identified based on various design choices for ground, underwater, and aerial scenarios. In addition, state-of-the-art methods for perception, planning, control, and interaction are elaborately discussed and their applicability in varied operational scenarios are presented. Then, some of the prominent methods are qualitatively compared, corresponding practicalities are illustrated, and their feasibility is analyzed for various use-cases. Furthermore, several prospective application areas are identified, and open problems are highlighted for future research.

1 Introduction

Person-following supports collaborative applications across industrial, health-care, surveillance, social, and entertainment settings, but its design varies across operating media and application constraints. The paper organizes this literature, analyzes state-of-the-art approaches, and identifies open problems.

  • The paper surveys autonomous person-following across manufacturing, warehousing, health care, surveillance, social interaction, and entertainment applications.
  • Ground, underwater, and aerial scenarios impose different constraints on sensors, robot construction, dynamics, and autonomy.
  • Person-following is vital to completing broader robot tasks even when following is not the robot’s primary objective.
  • The paper categorizes person-following behaviors, identifies operational and design issues, discusses perception, planning, control, and interaction methods, and qualitatively compares their feasibility.
  • It concludes by highlighting prospective applications and open problems for future research.

2 Categorization of Autonomous Person-Following Behaviors

The paper categorizes autonomous person-following by operating medium and additional application-specific attributes, then structures its discussion around those categories.

  • Person-following behaviors vary with operating medium, sensor choice, interaction mode, granularity, and degree of autonomy.
  • The operating medium is divided into ground, underwater, and aerial scenarios, while other attributes provide additional subdivisions.
  • Explicit interaction means direct human-robot communication, whereas implicit interaction means indirect communication.
  • Granularity denotes the number of humans and robots involved in a person-following scenario.
  • The paper’s simplified categorization is depicted in Figure 1 and organizes the remainder primarily by operating medium.

2.1 Ground Scenario

Ground person-following robots commonly combine visual and proprioceptive sensing with sensor fusion, while supporting implicit or explicit interaction and varying autonomy across applications.

  • Ground robots follow individuals or groups in domestic, industrial, health-care, and related assistant applications.
  • Most ground robots use cameras for perception, with additional sensors measuring distance, activities, relative motion, and orientation.
  • Multiple sensors and standard fusion techniques are used to reduce uncertainty in sensing and estimation.
  • Ground robots interact implicitly through spatial conduct and motion behavior or explicitly through voice, gestures, haptic interfaces, and smartphone applications.
  • Domestic applications often pair one robot with one person, whereas groups and multiple robots introduce more demanding coordination requirements.
  • Ground robots are typically fully autonomous, although task-specific and robot-guiding applications may use partial or semi-autonomous behavior.

2.2 Underwater Scenario

Underwater person-following supports cooperative missions but faces severe sensing, navigation, and autonomy constraints. Robots therefore combine vision with acoustic or inertial sensing and often retain explicit human interaction, especially in complex operations.

  • Underwater robots follow divers during cooperative tasks including shiphull and pipeline inspection, marine-species study, search-and-rescue, and surveillance.
  • Choice of sensors: Vision is preferred because acoustic bandwidth is limited and passive cameras avoid emitting energy, but marine conditions make visual tracking difficult.
  • Choice of sensors: Sonars and USBL complement vision by supporting rediscovery, global positioning, and more reliable tracking when visual or acoustic measurements are degraded.
  • Mode of interaction: Because fully autonomous underwater navigation remains an open challenge, explicit diver interaction is crucial for adjusting missions and supervising exploration.
  • Mode of interaction: Underwater missions have limited social and behavioral interaction, but implicit interaction remains vital for safety and can increase diver cognitive load.
  • Degree of autonomy: Semi-autonomous robots commonly accept human navigation inputs for simple exploration and data collection, while complex surveillance and monitoring require greater autonomy.

2.3 Aerial Scenario

Aerial person-following is shaped by short battery-limited episodes, lightweight sensing, and frequent user involvement. Partial autonomy suits controlled activities, whereas remote or critical missions demand more autonomous planning.

  • UAVs follow athletes for activities such as climbing or skiing, commonly operating in short episodes bounded by take-off, following, snapshot, and landing commands.
  • Choice of sensors: Weight, cost, size, and battery constraints favor front-facing cameras plus a downward optical-flow sensor in typical person-following UAVs.
  • Choice of sensors: Critical UAV applications can accommodate multiple high-resolution cameras, range sensors, stereo cameras, infrared sensors, ultrasonic sensors, and camera gimbals.
  • Choice of sensors: IMUs, optical flow, ultrasonic or pressure sensors, and GPS support pose estimation, stabilization, altitude measurement, and trajectory control.
  • Operational constraints: Frequent take-offs and landings require users to monitor UAV location, battery, and WLAN range during operation.
  • Granularity: Multiple UAVs can overcome single-platform limitations but substantially increase user cognitive load through coordination of batteries, positions, movements, and landings.
  • Degree of autonomy: Partial autonomy is preferred for many UAV applications, while fully autonomous planning is suited to remote surveillance, rescue, and suspect-following tasks.

3 State-of-the-art Approaches

State-of-the-art person-following systems organize perception around feature and model perspectives, then combine detection, tracking, learning, and sensor processing across operating media. Deep models offer strong visual performance but remain sensitive to training coverage and environmental changes.

  • Person-following systems comprise perception, planning, control, and interaction, with perception methods categorized by feature usage and prior target knowledge.
  • Ground Scenario: Ground robots generally fuse camera and other sensory inputs to localize people within a two-dimensional unicycle-model setting.
  • Feature-based Tracking: Feature-based methods detect person-specific patterns, while mean-shift and particle filters iteratively refine candidate locations through feature-space search.
  • Feature-based Learning: Background subtraction, tracking priors, HOG-SVMs, AdaBoost, and Bayesian models reduce search or learn person-location hypotheses from sensory features.
  • Feature or Representation Learning: Deep learning jointly learns feature representations and detection hypotheses, and CNN feature maps feed classifiers or regressors for person and object detection.
  • Feature or Representation Learning: CNN-based perception is highly accurate and robust to noise and illumination changes, but requires comprehensive training data and can degrade under occlusion, appearance changes, and environmental variation.

X Z Y

Person-following perception varies with operating medium, trajectory, viewpoint, and available computation. Ground, underwater, and aerial systems therefore use different sensing and tracking strategies, with aerial deep-learning deployment constrained by onboard resources.

  • Underwater scenario: Underwater tracking uses color, optical flow, sonar, motion-direction models, and CNNs, with deep models detecting multiple objects but remaining limited by slow onboard inference.Quantization and pruning can balance robustness and efficiency, while embedded supercomputing is making onboard deployment more feasible.
  • Aerial scenario: Aerial perception depends on trajectory and viewpoint: standard feature-based and pedestrian detectors suit close, smooth flights, whereas rapidly changing trajectories require perspective-aware sensing.Ground-plane estimation, 3D mapping, and target-motion prediction support more challenging outdoor applications.
  • Aerial scenario: Depth sensing improves aerial tracking reliability, but RGBD-based systems are limited to indoor environments, small motions, and often remote computation.The cited indoor system combines a regular camera for UAV localization with a depth camera for 3D person detection.
  • System overview: Person-following perception is illustrated as a processing pipeline that converts sensory data into target estimates for downstream robot operation.The generic flow is presented for autonomous object-following systems.
  • Aerial scenario: Deep learning-based person detectors remain underexplored for aerial applications because consumer UAVs have limited onboard computational resources.Faster mobile supercomputers and low-power computing may enable more common deployment.

3.2 Planning and Control

Planning and control transform estimated target states into obstacle-aware trajectories and navigation commands. Because person-following occurs in partially observable, dynamic environments, local online planning is generally required, while planning modality depends on mapping, sensing, and servoing choices.

  • Planning and control: The control pipeline estimates target state, refines it through filtering, generates obstacle-aware waypoints, optimizes trajectories, and feeds set-points to feedback controllers.Ground robots operate in 2D, while underwater and aerial robots operate in 3D; trajectory objectives can include travel time and safety.
  • Path planning: Person-following robots require local and online path planning to adapt to irregular and unpredictable changes in partially observable, dynamic environments.Planning is categorized by sensing locality, execution timing, mapping information, and algorithmic structure.
  • Mapping: Map-assisted planning suits structured environments with known maps, whereas target-centric planning builds partial maps from local sensing when global maps are unavailable.The distinction is especially relevant between indoor ground robots and outdoor aerial systems.
  • Position-based servoing: Position-based servoing plans from the robot’s estimated position to the person’s estimated position, using either map-assisted or target-centric information.Standard planners use occupancy representations and search methods, while probabilistic, heuristic, evolutionary, and POMDP approaches approximate or optimize paths.
  • Image-based servoing: Image-based servoing uses image features for path planning when accurate target localization is difficult, including underwater and GPS-denied settings.Diver-following robots may simplify this component by using bounding-box reactive planning or straight-line trajectories.
  • Socially aware planning: Crowded-area planning must incorporate dynamic obstacles, nearby human motion, and social norms such as passing conventions and walking speed.These requirements extend planning beyond geometric collision avoidance to socially aware behavior.

3.3 Interaction

Person-following interaction combines explicit communication channels with implicit behavioral expectations. These mechanisms vary by medium, distance, and application, while implicit interaction remains difficult to quantify and requires user studies.

  • Explicit interaction: Explicit interaction uses speech, markers, hand gestures, smart devices, and paired wearables to send operational instructions to the robot.Commands range from starting or stopping following to directional actions, recording, emergency landing, and procedural tasks.
  • Explicit interaction: Underwater marker-based communication is robust in noisy conditions because fiducial markers are easy to detect, but carrying many markers is inconvenient and unintuitive.RoboChat assigns marker sequences to symbolic grammar rules.
  • Explicit interaction: Aerial gesture communication is constrained by long, varying human-robot distances, so reliable gestures may first bring the UAV closer before other gestures are used.This addresses the difficulty of detecting small gestures from distant viewpoints.
  • Implicit interaction: Implicit interaction includes spatial, appearance-and-gaze, and motion conduct that define proximity, responsiveness, gaze tracking, following directions, turning, and waiting behaviors.These conduct rules inform safe trajectories, proximity control, motion prediction, and planning during interaction.
  • Implicit interaction: The technical characteristics of implicit interaction are difficult to quantify and require rigorous user studies and feasibility analysis.This limits how readily the behaviors can be formalized into research questions and solutions.

4 Qualitative Analysis: Feasibility, Practicality, and Design Choices

The paper compares person-following systems by sensing, computation, planning, interaction, and feasibility considerations across operating scenarios. It finds strong progress in ground systems but continuing needs in social behavior and underwater and aerial applications.

  • Detection and tracking: Perception design trades detection and tracking performance against sensor availability, onboard computation, and power consumption.Sensor fusion supports fast accurate tracking, deep models demand more computation, and UWB/RFID tags can support low-power designs.
  • Planning and control: Planning and control choices depend on autonomy, dynamic obstacles, environmental knowledge, and social constraints.Static settings may rely on the companion for collision-free navigation, whereas crowded and outdoor settings require explicit obstacle and human-motion handling.
  • Overall feasibility: Current state-of-the-art systems provide very good solutions for the many research problems addressed in ground person-following.The qualitative comparison covers systems discussed across perception, planning, control, and interaction.
  • Future directions: Social and behavioral aspects need more attention, while underwater and aerial systems require smarter methods for their unique operational challenges.These priorities are identified as future research directions after comparing interactivity and general feasibility.

5 Prospective Research Directions

Prospective person-following research must address team tracking, nontraditional relative positions, and anticipative behavior across demanding operational settings.

  • 5 Prospective Research Directions: Following teams in 3D environments requires coordinated perception, motion planning, control, and interaction under real-time constraints.Detecting multiple people is relatively straightforward, but tracking independently moving team members and defining spatial conduct remain challenging.
  • Following a Team: Team-following is potentially valuable for underwater missions and UAV-based coverage of social or sports events despite its operational difficulty.These applications involve teams of divers or independently moving people in three-dimensional environments.
  • Following Behind or Ahead?: Robot-following applications increasingly require positioning ahead of or beside the person rather than consistently following from behind.Shopping carts may stay ahead, while sports-filming UAVs may move around the person for better viewpoints.
  • Following Behind or Ahead?: Anticipative following requires predicting human trajectories and recovering from incorrect predictions or actions.Motion history, gaze behavior, and prior knowledge about the environment or destination can support anticipative behavior.

5.4 Learning to Follow from Demonstration

Learning robot behavior from demonstrations is a promising direction for person-following, while communication, social norms, and long-term interaction remain important practical concerns.

  • 5.4 Learning to Follow from Demonstration: End-to-end learning from demonstration could simplify autonomous person-following, but broader real-world validation remains necessary.Existing results include navigation and driving, while person-following studies have largely remained in simulation.
  • Human-Robot Communication: Human-robot dialogue needs methods for robot-initiated communication and risk assessment when interactive interfaces are unavailable.Speech, markers, and hand gestures are discussed, but robot-initiated dialogue is comparatively underexplored.
  • Social Interaction: Socially deployed robots should account for preferred spatial and motion behaviors, including anticipatory actions such as leaving space for door opening.These behaviors are connected to social norms in person-following settings.
  • Long-Term Interaction: Long-term interaction with companion robots has potential applications in learning, therapy, and physical exercise.The cited studies report benefits for child-robot interaction and exercise support.

5.8 Specific Person-Following

Specific-person following is important in crowded and assistive settings, but robust re-identification and embedded computation remain practical challenges for autonomous robots.

  • 5.8 Specific Person-Following: Specific-person following is especially useful in multi-human, crowded, social, elderly-care, and disability-support settings.Face or body-pose recognition can provide an additional identification module in some applications.
  • 5.9 Person Re-identification: Existing person-following systems use template matching and trajectory replication for recovery, while newer appearance-based re-identification models remain underused.Applying these deep models to human-dominated social settings is identified as a research direction.
  • 5.11 Embedded Parallel Computing Solutions: Deep learning improves perception robustness but remains computationally expensive and dependent on parallel computing platforms.Embedded systems must also address power consumption, durability, and cooling.
  • 5.8 Specific Person-Following: Privacy and safety concerns constrain person-following UAV deployment in social and public environments.The cited survey reports that about 54% of the US population opposed drones flying near people’s homes.

6 Conclusion

The paper categorizes person-following across operational media and design choices, reviews methods and practical constraints, and identifies open problems for future research.

  • 6 Conclusion: The review covers person-following design issues and operational challenges across ground, underwater, and aerial scenarios.It also discusses state-of-the-art perception, planning, control, and interaction methods.
  • 6 Conclusion: Qualitative analysis compares operational considerations, assumptions, and feasibility across different use cases.The paper connects algorithmic approaches with their practical deployment conditions.
  • 6 Conclusion: Open problems and potential applications are highlighted as directions for strengthening the literature and bridging research with practice.The conclusion frames future progress around improved solutions to these identified challenges.
Loading 1803.08202v4…