Source-linked AI summary
Vision-Based Leader-Follower Formation Control for Cooperative UAVs in GPS-Degraded Environments
Deekshitha Angadi, Naveena Budda, Vikas Agarwal, Rojesh Arunkumar Mulasa, Ravi Killamsetty, Mohamed Samshad, Narsimlu Kemsaram
TL;DR
Cooperative UAV formations need reliable relative perception when GPS or other absolute positioning degrades. This paper develops an onboard RGB-D and YOLOv8 framework that estimates leader pose for formation control and fallback operation, and evaluates it in XTDrone simulation, where it maintains stable formations with reasonable computational cost.
Problem
Absolute positioning assumptions are fragile in cluttered or GPS-denied environments, creating a need for follower onboard perception of neighbours’ relative pose.
Method
The follower uses a trained YOLOv8 detector, RGB-D depth, pinhole-camera geometry, temporal filtering, and ROS-integrated control to estimate the leader’s relative pose.
Results
The framework demonstrates promising detection, relative-pose accuracy, formation-keeping behaviour, and computational cost, including fallback operation when absolute positioning sensors fail.
Takeaways & Limitations
Vision-based relative pose provides a viable fallback for maintaining cooperative UAV formations when GPS is degraded or unavailable.
Abstract
from arXiv · showhide
Cooperation in multi-UAV systems requires reliable relative perception so that follower vehicles can maintain formation and continue their mission safely even when absolute positioning sensors degrade or fail. This paper presents a vision-based cooperative formation framework running on a follower UAV that uses a front-facing RGB-D camera to detect, track, and localize a leader UAV in real-time. A lightweight YOLO-based detector is trained on a dedicated drone dataset and deployed onboard to predict leader bounding boxes, which are then fused with depth information via a pinhole camera model to estimate the leader's relative pose. These estimates provide a leader-follower position controller and can also be used as a backup when GPS or external localization is unavailable. This framework is implemented as a set of ROS nodes and evaluated in a physics-based multi-UAV simulation built on XTDrone, with sensor noise and communication dropouts. We evaluate detection accuracy, runtime, and formation-keeping error under nominal conditions and under simulated failures of the positioning sensors. The results show that the proposed framework maintains stable leader-follower formations with reasonable computational cost and provides a practical basis for extending vision-based cooperative formation control to real-world multi-UAV systems.
I. Introduction
GPS and motion-capture dependence is fragile in cluttered or GPS-denied environments, motivating onboard relative perception for cooperative UAV formations. The paper explores an RGB-D, vision-based leader-follower framework that estimates relative pose and supports formation control and positioning-sensor failure recovery.
- Absolute positioning systems such as GPS or MoCap can be fragile in cluttered or GPS-denied environments.
- Follower UAVs therefore require local onboard perception to estimate neighbours’ relative pose using their own sensors.
- The framework studies a minimal one-leader, one-follower formation in which the follower uses a front-facing RGB-D camera.
- A lightweight perception pipeline detects the leader and reconstructs relative pose from depth using a calibrated pinhole camera model.
- The relative pose estimate maintains formation and provides a safety layer when positioning sensors fail.
- The paper contributes a modular YOLOv8 and RGB-D framework, ROS–XTDrone integration, and evaluation under nominal and sensor-failure conditions.
A. Cooperative UAVs and Leader-Follower Formation
Prior work establishes leader-follower control and vision-based UAV localization, while deep-learning detectors support markerless drone recognition. The paper positions its contribution as an integrated detector, RGB-D localization, and formation-control framework.
- A. Cooperative UAVs and Leader-Follower Formation: Leader-follower formation is a widely adopted coordination pattern for cooperative UAV missions.
- A. Cooperative UAVs and Leader-Follower Formation: Classical formation-control approaches include virtual-structure, behaviour-based, and graph-theoretic methods.
- B. Vision-Based Relative Localization and Formation: Vision-based relative localization suits UAVs because cameras are lightweight, low-cost, and provide inter-agent scene information.
- B. Vision-Based Relative Localization and Formation: Fiducial markers can provide accurate short-range 6-DoF pose but require clear line-of-sight and environmental information.
- B. Vision-Based Relative Localization and Formation: Markerless localization instead uses natural imagery with feature matching and epipolar geometry.
- C. Deep-Learning-Based Drone Detection: YOLO, Faster R-CNN, and SSD detectors recognize UAVs without fiducial markers across varied scales, backgrounds, and viewing angles.
- C. Deep-Learning-Based Drone Detection: The paper bridges these areas by integrating deep-learning detection, RGB-D sensing, and leader-follower control in ROS and XTDrone.
III. Proposed Vision-Based Leader-Follower Formation for Cooperative UAVs
The proposed system uses a two-UAV leader-follower setting in which the follower estimates the leader’s relative pose onboard. Its architecture combines RGB-D perception, YOLOv8 detection, temporal filtering, and controller input, with vision fallback during positioning failure.
- III. Proposed Vision-Based Leader-Follower Formation for Cooperative UAVs: The study considers two multirotor UAVs: a leader following a predefined trajectory and a follower maintaining a desired relative offset pose.
- III. Proposed Vision-Based Leader-Follower Formation for Cooperative UAVs: Conventional approaches infer relative pose primarily from GPS or MoCap systems.
- III. Proposed Vision-Based Leader-Follower Formation for Cooperative UAVs: The follower instead relies on onboard camera and computation when absolute positioning information is degraded or lost.
- III. Proposed Vision-Based Leader-Follower Formation for Cooperative UAVs: The architecture enables normal formation maintenance and vision-only fallback when positioning sensors fail.
- III. Proposed Vision-Based Leader-Follower Formation for Cooperative UAVs: The fallback can support mission continuation or safe landing.
- A. Training Dataset: The detector dataset combines public UAV images with rendered PX4 drone views across altitudes, angles, distances, and static or flying configurations.
- A. Training Dataset: Approximately 1,339 annotated images were divided into 80% training and 20% testing splits without separating short temporal sequences.
B. Drone Detection
Drone detection uses a compact YOLOv8 detector on RGB images, while aligned depth and pinhole-camera geometry convert the detected image location into the leader’s relative 3D position.
- The compact YOLOv8 detector outputs bounding boxes, objectness scores, and class probabilities, then applies non-maximum suppression.
- The pipeline retains the highest-confidence bounding box for the “drone” class when its confidence exceeds a fixed threshold.
- The RGB-D camera supplies synchronized RGB and aligned depth data for each frame.
- The pixel-center coordinates (u, v) and local depth Z are transformed with intrinsic matrix K through the pinhole camera model into 3D point P(X, Y, Z).
D. Temporal Filtering and Fusion
The framework filters noisy vision-based relative pose estimates before using them in a simple leader–follower position-control pipeline. The filtered pose supports formation-error computation and velocity commands, including fallback operation when follower positioning fails.
- Filtering: A constant-velocity Kalman filter smooths relative position estimates affected by depth noise and missed detections.The filter predicts the state forward and corrects it when new vision measurements arrive.
- Fallback operation: When the follower’s positioning sensor fails, the formation controller relies primarily on the estimated vision-based relative pose.This design provides the fallback behavior evaluated under positioning-sensor failure.
- Pose representation: The vision pipeline estimates the leader’s position in the follower’s camera frame using longitudinal, lateral, and vertical offsets.Z denotes optical-axis distance, while X and Y denote lateral and vertical offsets.
- Formation target: A desired target offset defines the leader’s intended relative position, such as centered in the image at a fixed distance.
- Position control: The controller computes position error relative to the target and converts it into a follower velocity command using a decoupled PID law.The gains are diagonal matrices, and the resulting command is sent in PX4’s local reference frame as an offboard velocity setpoint.
- System integration: The ROS architecture publishes estimated relative pose from the perception pipeline to the leader–follower position controller via MAVROS.
IV. Implementation
The perception framework is implemented as modular ROS nodes that connect RGB-D sensing, YOLOv8 detection, and pinhole-model pose estimation.
- ROS implementation: Each logical component runs as a ROS node, enabling substitution and experimentation across the perception pipeline.
- Sensing and detection: The camera node publishes aligned color and depth images, while the detection node predicts leader bounding boxes and confidence scores from RGB images.
- Pose estimation: A pose-estimation node combines detected bounding boxes with depth images and applies the pinhole model to estimate relative pose.
V. Evaluation
The framework is evaluated in an XTDrone multi-UAV simulation integrating PX4 SITL, ROS, and Gazebo. The evaluation includes visual tracking scenarios, noisy measurements, and four performance cases.
- Simulation setup: The XTDrone simulation integrates PX4 SITL, ROS, and Gazebo in an outdoor environment with textured ground and visual structures.
- Simulation setup: Two quadrotors are instantiated: a leader follows predefined straight-line or circular trajectories, while the follower switches to leader–follower mode after detection.
- Sensor conditions: Gaussian noise and biases are added to simulated measurements to represent real-world sensor conditions.
- Evaluation scenes: The simulation scenes include leader–follower views, the follower camera view, onboard bounding-box detection, and an outdoor tracking scenario.
- Evaluation cases: The evaluation covers detection performance, relative pose accuracy, formation-keeping behavior, and computational cost.
A. Detection Performance
The detector achieves high precision and recall on held-out drone data and maintains a stable leader lock during most simulated trajectories. Dropouts occur mainly when the leader leaves the field of view or becomes very small.
- Held-out detection: The YOLOv8-based detector achieves high precision and recall across typical viewing angles and ranges on the held-out drone dataset.
- Failure cases: Short detection dropouts occur primarily during rapid turns or large separations, when the leader exits the field of view or appears very small.
- Temporal filtering: A temporal filter mitigates many dropout effects by interpolating relative pose between successful detections.
B. Relative Pose Accuracy
Relative-pose accuracy remains generally acceptable during nominal trajectories and after GPS loss, but degrades during aggressive manoeuvres and at larger distances.
- Relative position error remains below one metre and yaw error within five degrees for most of the trajectory under nominal conditions.These values are reported against simulator ground truth using RMSE estimates.
- Errors increase during aggressive manoeuvres and at larger distances because of depth-sensing and camera-resolution limitations.
- Formation error briefly increases when GPS is disabled before settling at a slightly higher but still acceptable level.The controller then relies fully on the vision-based relative pose.
D. Computational Cost
The perception pipeline operates at approximately 25 Hz with moderate CPU and GPU utilization in the simulation environment, indicating reasonable computational cost.
- 38 milliseconds per frame corresponds to an update rate of roughly 25 Hz in the simulation environment.
- Moderate CPU and GPU utilization suggests potential deployment on low-power onboard companion computers with modest optimization.The paper identifies platforms including Nvidia Jetson, Raspberry Pi, and Intel NUC devices.