PaperlayerView exact revision on arXiv

Source-linked AI summary

Decentralized Visual-Inertial-UWB Fusion for Relative State Estimation of Aerial Swarm

Hao Xu, Luqi Wang, Yichen Zhang, Kejie Qiu, Shaojie Shen

arXiv:2003.05138v1cs.RO

TL;DR

Aerial swarms lack a decentralized relative-state estimation method, while existing approaches rely on environmental conditions or external positioning infrastructure. This paper combines VIO, visual detection, and UWB measurements in a decentralized onboard framework; experiments report centimeter-level precision that outperforms UWB- and vision-based methods.

  • Problem

    Aerial swarms lack a state-of-the-art decentralized relative-state estimator, while existing positioning approaches have environmental or external-infrastructure constraints.

    Source: §I · p. 1

  • Method

    The framework uses a two-stage optimization process, first estimating VIO from stereo-camera and IMU data, then fusing VIO, detection, and UWB relative measurements.

    Source: §III · p. 2

  • Results

    Experiments against motion-capture ground truth report centimeter-level precision, outperforming UWB- and vision-based methods.

    Source: §VI · p. 6

  • Takeaways & Limitations

    The fully decentralized system is not limited by camera field of view or GPS and has potential for aerial-swarm applications in different environments.

    Source: B. Experiment Result and Comparison · p. 6

Abstract

from arXiv · show

The collaboration of unmanned aerial vehicles (UAVs) has become a popular research topic for its practicability in multiple scenarios. The collaboration of multiple UAVs, which is also known as aerial swarm is a highly complex system, which still lacks a state-of-art decentralized relative state estimation method. In this paper, we present a novel fully decentralized visual-inertial-UWB fusion framework for relative state estimation and demonstrate the practicability by performing extensive aerial swarm flight experiments. The comparison result with ground truth data from the motion capture system shows the centimeter-level precision which outperforms all the Ultra-WideBand (UWB) and even vision based method. The system is not limited by the field of view (FoV) of the camera or Global Positioning System (GPS), meanwhile on account of its estimation consistency, we believe that the proposed relative state estimation framework has the potential to be prevalently adopted by aerial swarm applications in different scenarios in multiple scales.

I. INTRODUCTION

Aerial swarms need real-time relative state estimates for coordinated flight, but existing GPS, motion-capture, and UWB approaches depend on environmental conditions or external infrastructure. The paper proposes onboard vision-UWB estimation in response to limitations in prior methods.

I. INTRODUCTION

  • GPS methods are restricted to outdoor use and provide meter-scale accuracy, while indoor motion-capture and UWB systems require external cameras or ground anchors.RTK-GPS can achieve centimeter-level positioning but requires a ground station.Source: §I · p. 1
  • VIO works in indoor and outdoor, known and unknown environments, but local-frame estimates drift and camera-based detection cannot estimate vehicles outside the camera’s field of view.The paper motivates adding an omnidirectional onboard sensor such as UWB.Source: §I · p. 1
  • The proposed system performs onboard relative-state estimation using vision and UWB, with each vehicle estimating individually and sharing information without offboard sensors or computing.The paper also claims robustness to missing detections and individual vehicle malfunctions.Source: §I · pp. 1–2

II. RELATED WORKS

  • Prior UWB and vision-based relative-localization approaches have application limits, including 2-D-only estimates, challenging initialization, precision concerns, or dependence on visual markers remaining in view.The related-work discussion describes leader-follower formations as a limit of marker-based systems.Source: §II · p. 2

III. RELATIVE STATE ESTIMATION FRAMEWORK

The framework uses a two-stage estimation process: VIO supplies motion estimates, then detection and UWB relative-distance measurements are fused to estimate the swarm’s relative states. The resulting estimates feed back into detection and tracking.

III. RELATIVE STATE ESTIMATION FRAMEWORK

  • The estimator uses a detector and tracker for visible drones and incorporates UWB distance and VIO measurements to optimize relative translation and yaw.The estimated state has four degrees of freedom: x, y, z, and yaw rotation ψ.Source: §III · p. 2
  • The framework first estimates visual-inertial motion from stereo-camera and IMU data, then fuses VIO, detection, and UWB relative measurements through optimization.The relative estimates are sent back to facilitate detection.Source: §III · p. 2

A. Notations

  • The notation distinguishes estimated states, time-indexed measurements, and states expressed in a drone’s body frame or VIO frame.The supplied nomenclature also identifies relative-pose and inter-drone distance notation.Source: A. Notations · p. 2
  • The notation includes the transformation between consecutive times from each drone’s VIO result and the distance between drones at a given time.These quantities are used alongside the relative-state and measurement notation.Source: A. Notations · p. 2

B. Problem Formulation

For each drone k, the problem is to estimate the pose of every other drone i in k’s VIO frame at time t. The formulation uses drone k’s VIO-estimated ego motion to retrieve the relative pose.

B. Problem Formulation

  • For an arbitrary drone k, estimate each drone i’s pose in drone k’s VIO frame at time t.The formulation considers a swarm containing N drones.Source: B. Problem Formulation · p. 2
  • Drone k’s VIO-estimated ego motion is used to retrieve drone i’s pose in drone k’s VIO frame.This expresses the relative pose in the reference drone’s VIO frame.Source: B. Problem Formulation · p. 2

C. The First Stage: VIO

The first stage estimates each drone’s visual-inertial motion from stereo-camera images and IMU data. Its VIO output is treated as a measurement in the relative-state estimator.

C. The First Stage: VIO

  • The first stage uses GPU-accelerated VINS-Fusion to estimate visual-inertial motion from stereo-camera images and IMU data.The tightly coupled VIO result is passed as a measurement to the second stage.Source: C. The First Stage: VIO · pp. 2–3
  • Onboard sensor data are processed and broadcast to other terminals, while the relative-state estimator combines VIO, detection, and UWB-distance information.The resulting estimates also serve planning and control.Source: Figure 2 · p. 3
  • The VIO produces a time-indexed transformation-matrix measurement modeled with Gaussian noise.The passage states that this measurement is frame invariant.Source: C. The First Stage: VIO · p. 3
  • The VIO is also used for the system’s planning and control purposes.This is in addition to its role as a measurement for relative-state estimation.Source: C. The First Stage: VIO · p. 3

D. Second Stage

The second-stage optimization combines VIO with relative measurements from detection and tracking and UWB. Detection and tracking recover 3-D relative positions, while matching assigns detected drones unique IDs.

1) Detection and Tracking of Aerial Platforms:

  • Detection and tracking use YOLOv3-tiny, MOSSE tracking, depth, and camera geometry to recover 3-D relative positions in the observing drone’s body frame.The detector is trained on labeled images of the customized platforms; bounding-box centers provide depth-camera measurements.Source: 1) Detection and Tracking of Aerial… · p. 3
  • Bounding-box size provides a second depth cue, and detections whose reprojected size differs greatly from the drone’s real size are treated as outliers.Source: 1) Detection and Tracking of Aerial… · p. 3
  • Because the drones look alike, detections are assigned unique IDs by matching visual positions against the framework’s estimates, with disparities above a threshold rejected as outliers.Detector and tracker boxes with the same ID must overlap sufficiently; otherwise matching is repeated.Source: 1) Detection and Tracking of Aerial… · p. 3

D. Second Stage

2) UWB distance measurements:

Frames combine VIO with relative UWB-distance and detection-and-tracking measurements. Keyframes are selected by new detections or motion, and chronological keyframes form a bounded optimization window.

2) UWB distance measurements:

  • Each frame contains measurements and VIO at a specific time, including UWB pairwise distances, VIO ego-motion results, and detection-and-tracking relative translations.Source: 3) Data Frame, Keyframe Selection and… · pp. 3–4
  • The sliding-window illustration distinguishes VIO pose transformations, UWB relative distances, and detection-and-tracking relative 3-D translations by arrow color.Source: Figure 3 · p. 4
  • A frame becomes a keyframe when it contains new drones or the drone moves beyond threshold td0 from its previous keyframe.Source: 3) Data Frame, Keyframe Selection and… · p. 4
  • A drone moving beyond threshold td1 and detecting, or being detected by, another drone also triggers a keyframe; chronological keyframes form a sliding window capped at 50 frames.The adjustable thresholds typically satisfy 0 < td1 < td0.Source: 3) Data Frame, Keyframe Selection and… · p. 4

4) Optimization-based Relative State Estimation:

Relative states are estimated by optimizing a bundle-adjustment formulation over a sliding window of drone states and measurement residuals. The solution combines distance, detection, and VIO information, with real-time VIO supporting higher-rate pose predictions.

4) Optimization-based Relative State Estimation:

  • For drone k, the full state vector is optimized over the sliding window, and the VIO residual maintains local consistency between the drones’ VIO frames.Solving the optimization provides the relative state estimate.Source: 4) Optimization-based Relative State… · p. 4
  • The bundle-adjustment formulation minimizes Mahalanobis norms of distance, detection, and VIO residuals to obtain a maximum a posteriori estimate.The measurement sets are denoted L for distances, D for detections, and S for VIO ego motion.Source: 4) Optimization-based Relative State… · p. 4
  • The authors use Ceres with a trust-region sparse normal Cholesky method and compose optimized relative poses with real-time VIO for 100 Hz predictions.Source: 4) Optimization-based Relative State… · p. 4

5) Residual and Variable Pruning:

Hovering-based pruning reduces the optimization workload while avoiding potential overfitting from noisy distance measurements. In a five-drone indoor case, average optimization time fell substantially on the onboard computer.

5) Residual and Variable Pruning:

  • The formulation has 4mn variables; when two drones hover, distance residuals involving another drone can be pruned because noisy ranges may overfit triangulation.Source: 5) Residual and Variable Pruning · pp. 4–5
  • 64.3% fewer residual elements and 50% fewer variables yield a 4× optimization speedup after pruning.The reported pruning fixes hovering-drone poses and removes their VIO residuals.Source: 5) Residual and Variable Pruning · p. 5
  • 686.6 ms to 169.6 ms: average optimization time in a five-drone indoor flight on a DJI Manifold 2-G; with negligible VIO drift over that period, the algorithm is treated as real-time.Source: 5) Residual and Variable Pruning · p. 5

6) Estimator Initialization:

Initialization addresses a highly nonlinear, initially unobservable relative-pose problem. It assumes small yaw differences at startup, then uses repeated trials to initialize translation for sufficiently moving drones.

6) Estimator Initialization:

  • Initialization assumes observed drones initially face similar directions because relative yaw is unobservable until they have moved sufficiently.The assumption treats yaw differences as small; after sufficient movement, random translational initializations are tried and the lowest-cost result is selected before normal optimization.Source: 6) Estimator Initialization · p. 5

IV. DETAILED SYSTEM IMPLEMENTATION

The system runs estimation and planning onboard and tests relative poses against motion-capture ground truth in indoor swarm flights. Reported results include centimeter-level errors and comparisons with prior UWB- and vision-based methods.

A. Aerial Platform

  • Each of five identical platforms combines a D435i stereo camera, embedded-IMU DJI N3, UWB module, and onboard Manifold2-G; IMU and images run at 400Hz and 20Hz.Estimation and planning run on the individual platforms; the ground station sends commands and monitors flight status, and the system uses ROS.Source: A. Aerial Platform · p. 5, Figure 4 · p. 5, §IV · p. 5
  • UWB modules broadcast synchronized communications at 100Hz between drones and to the ground station.The flight controller handles attitude control; other computations, including position control, state estimation, and trajectory planning, run onboard.Source: A. Aerial Platform · p. 5

A. Experiment Setup

B. Experiment Result and Comparison

  • Fig. 5 shows consistent real-time relative estimates between drone 0 and the other drones in a five-drone formation flight.The reciprocal estimates are inverted for comparison.Source: Figure 5 · p. 6, B. Experiment Result and Comparison · p. 6
  • 6.84cm translation and 2.131 degree yaw RMSE were reported over a 34.553m two-drone flight.The trajectory comparison aligns the estimated and ground-truth frames for visualization.Source: Figure 6 · p. 6
  • The proposed method outperforms ’s 0.14m horizontal and 0.28m total RMSE and ’s 0.2m horizontal error, and improves on ’s 10–20cm vision error while handling out-of-FoV vehicles.The paper reports that does not provide a vertical error.Source: B. Experiment Result and Comparison · p. 6
  • 5cm-level RMSE in aggressive two-drone flight was similar to the triple-drone result.The comparison uses ground-truth tables for double- and triple-drone settings.Source: B. Experiment Result and Comparison · p. 5, Table I · p. 6, Table II · p. 6

VI. CONCLUSION AND FUTURE WORK

The paper presents a decentralized visual-inertial-UWB framework for relative state estimation, tested in aerial-swarm flights with up to five drones. It reports centimeter-level precision against motion-capture ground truth and identifies future work on safe flights in cluttered environments and collaborative search and rescue.

VI. CONCLUSION AND FUTURE WORK

  • The framework combines detection, VIO, and distance measurements through optimization for decentralized relative-state estimation.The authors report centimeter-level precision against motion-capture ground truth, outperforming the UWB- and vision-based methods they compare against.Source: §VI · p. 6
  • Future work will extend the system toward safe flights in cluttered environments and collaborative search-and-rescue tasks.The authors also identify control and planning algorithms for aerial swarms as possible developments based on the relative-estimation framework.Source: §VI · p. 6
Loading 2003.05138v1…