PaperlayerView exact revision on arXiv

Source-linked AI summary

Omni-swarm: A Decentralized Omnidirectional Visual-Inertial-UWB State Estimation System for Aerial Swarms

Hao Xu, Yichen Zhang, Boyu Zhou, Luqi Wang, Xinjie Yao, Guotao Meng, Shaojie Shen

arXiv:2103.04131v5cs.RO

TL;DR

Decentralized aerial-swarm state estimation remains difficult because existing approaches suffer from observability, initialization, accuracy, and global-consistency problems. Omni-swarm combines omnidirectional visual-inertial-UWB sensing with map-based localization, drone tracking, and graph optimization. Experiments report centimeter-level relative accuracy with global consistency and demonstrate inter-drone collision avoidance without external devices.

  • Problem

    Decentralized aerial-swarm state estimation must support relative states without centralized infrastructure, while prior methods face restricted observability, complicated initialization, insufficient accuracy, and missing global consistency.

  • Method

    Omni-swarm runs independently on each drone and fuses omnidirectional visual-inertial-UWB, map-based, and tracking measurements through graph-based optimization.

  • Results

    The system achieves centimeter-level relative state estimation accuracy while ensuring global consistency in aerial-swarm flight experiments.

  • Takeaways & Limitations

    Omni-swarm supports formation flights in complex environments and enables inter-drone collision avoidance without external devices.

  • Takeaways & Limitations

    The back-end limits the number of swarm keyframes to a preset maximum and randomly deletes older keyframes as the graph grows.

Abstract

from arXiv · show

Decentralized state estimation is one of the most fundamental components of autonomous aerial swarm systems in GPS-denied areas yet it still remains a highly challenging research topic. Omni-swarm, a decentralized omnidirectional visual-inertial-UWB state estimation system for aerial swarms, is proposed in this paper to address this research niche. To solve the issues of observability, complicated initialization, insufficient accuracy, and lack of global consistency, we introduce an omnidirectional perception front-end in Omni-swarm. It consists of stereo wide-FoV cameras and ultra-wideband sensors, visual-inertial odometry, multi-drone map-based localization, and visual drone tracking algorithms. The measurements from the front-end are fused with graph-based optimization in the back-end. The proposed method achieves centimeter-level relative state estimation accuracy while guaranteeing global consistency in the aerial swarm, as evidenced by the experimental results. Moreover, supported by Omni-swarm, inter-drone collision avoidance can be accomplished without any external devices, demonstrating the potential of Omni-swarm as the foundation of autonomous aerial swarms.

I. INTRODUCTION

Omni-swarm addresses decentralized aerial-swarm state estimation, where restricted observability, difficult initialization, limited accuracy, and missing global consistency hinder practical deployment. Its omnidirectional visual-inertial-UWB front-end and decentralized fusion framework target these issues while supporting autonomous swarm operations.

  • Motivation: External motion-capture, anchored-UWB, GPS, and RTK-GPS systems limit real-world deployment through centralized operation and bulky infrastructure.Such systems typically fail to meet the practical requirement of simple deployment.
  • Motivation: Decentralized aerial-swarm state estimation is needed for autonomous operation but is more complicated than single-drone estimation because each drone must estimate both ego and swarm states.State estimation supports higher-level functions such as path planning and mapping.
  • Challenges: Previous methods face restricted-FoV observability, complicated initialization, 20 cm–50 cm position errors, and lack of global consistency.Restricted observability occurs when other drones leave the camera field of view, while large initialization motions can create safety risks.
  • Approach: Omni-swarm introduces two 235° wide-FoV fisheye cameras, VINS-Fisheye, multi-drone map-based localization, and visual drone tracking within an omnidirectional perception front-end.Map-based localization addresses initialization and global consistency, while omnidirectional cameras address restricted observability.
  • Approach: The decentralized system runs independently on each drone, fuses front-end measurements through graph-based optimization, and uses outlier rejection for accurate and robust swarm-state estimation.Information is processed onboard rather than through a central server.
  • Capabilities: Omni-swarm enables non-line-of-sight estimation after revisiting the same place, global consistency, temporary drone joining or exiting, and robustness to temporary signal loss or partial sensor failures.The paper evaluates these capabilities through simulation, real-world flight experiments, and inter-drone collision avoidance.

D. Visual-inertial-UWB Fusion with Global Consistency

The visual-inertial-UWB fusion design combines onboard and broadcast measurements while addressing the infrastructure and line-of-sight constraints of prior global-localization approaches. Omni-swarm distributes estimation across onboard computers and maintains a graph of swarm information for real-time state estimation.

  • Prior limitations: Prior visual-inertial-UWB methods use fixed UWB anchors to eliminate odometry drift but require additional ground infrastructure and anchor line of sight.These requirements limit their real-world practical value.
  • System architecture: Each onboard computer collects UWB relative distances, VIO, map-based measurements, and detection results from onboard and broadcast information.The resulting real-time estimates support detection matching, planning, and control.
  • Global consistency: Omni-swarm guarantees global consistency without external infrastructure, making the system more flexible for real-world applications.Map-based localization is the mechanism identified for maintaining global consistency.
  • State representation: The swarm keyframe contains the keyframes of all drones, while the estimation graph stores information used for swarm-state estimation.The notation defines available drones as those with functioning VIO and stable network connections, including the estimating drone itself.

B. State Estimation Problem of Aerial Swarm

The aerial-swarm state-estimation problem requires each drone to estimate its own motion and the states of other drones in its local frame. VIO supplies ego-motion and gravity-based attitude information, while relative estimates support full pose reconstruction.

  • Problem formulation: For every drone k, the problem is to estimate the 6-DoF pose of each available drone i in drone k’s local frame at time t.The swarm may contain up to n homogeneous drones.
  • Problem formulation: The task separates into estimating drone k’s ego-motion state and estimating another drone i’s local and 4-DoF relative states.These two components define the state variables required by the formulation.
  • State reconstruction: VIO uses IMU-measured gravitational acceleration to help extract roll and pitch angles from attitude.Gravity is consistent among drones, supporting combination of ego-motion and relative-pose estimates.
  • State reconstruction: Combining a drone’s VIO estimate with its estimated 4-DoF pose produces the drone’s 6-DoF pose.The formulation uses the relative pose and local-frame pose to reconstruct the full state.
  • State reconstruction: The resulting transformation eliminates the yaw drift estimated by VIO.The yaw correction is represented using a rotation matrix about the z axis.

C. Global Consistency of the State Estimation

Omni-swarm targets drift-free swarm estimation by combining map-based localization with decentralized graph optimization. Its observability-aware design initializes and estimates only observable drones while integrating multiple relative measurements.

  • Map-based localization compensates for VIO’s long-term drift while estimating both ego-motion and relative swarm states.
  • Observability and initialization: Omni-swarm tracks each drone’s observability and initializes or estimates only drones whose states are observable.
  • Graph-based optimization: The back-end independently runs on each drone, preprocessing VIO, visual tracking, and map-localization measurements before graph-based state optimization.
  • Graph-based optimization: The factor graph uses swarm keyframes as vertices and measurements as edges, with ego-motion, distance, visual-detection, and map-based factors connecting poses.
  • Graph-based optimization: Distance factors use UWB measurements, while visual-detection factors provide accurate relative-state constraints between detected drones.

IV. FRONT-END: OMNIDIRECTIONAL PERCEPTION SYSTEM AND MEASUREMENT MODELING

The front-end provides omnidirectional visual-inertial perception and measurements for swarm estimation. It combines fisheye-camera VIO, distortion-free image reprojection, visual drone tracking, and relative-pose estimation.

  • Two 235-degree-FoV fisheye cameras provide surrounding-area coverage, while VINS-Fisheye estimates ego-motion using stereo images and IMU measurements.
  • Raw fisheye images are reprojected into five distortion-free images for subsequent visual processing.
  • VIO supplies real-time local pose and velocity estimates, while the back-end fuses its 4-DoF relative poses instead of original odometry because of long-term drift.
  • The VIO relative-pose noise is modeled as Gaussian, and keyframes retain distortion-free stereo images and camera parameters for later processing.
  • Visual drone tracking: VDT tracks detected drone targets, maintains anonymous or identified target files, and estimates their relative poses.
  • Visual drone tracking: The visual pipeline detects drone bounding boxes on distortion-free fisheye images using YOLOv4-tiny and represents each candidate by box-corner coordinates.

2) Visual Object Tracking of Drones:

Visual drone tracking combines efficient temporal tracking with data association and CNN-based relative-pose estimation. It links image detections to estimated drones or anonymous targets for subsequent swarm estimation.

  • MOSSE tracking maintains visual target files between detections, reducing the need to run computationally expensive detection at every frame.
  • New detections replace an existing target’s tracker when associated, or create a new visual target file when no association exists.
  • Data Association: Data association is essential in homogeneous swarms because detections must be matched with drones observed by state estimation.
  • Data Association: The unified association framework supports partially initialized systems and tracks anonymous targets alongside estimated drones.
  • Data Association: Overlap between bounding boxes defines association cost, and the Hungarian algorithm assigns candidates while accommodating missing detections and false alarms.
  • A CNN-based 6-DoF pose estimator extracts relative poses from semantic image features, which become visual-detection measurements in the back-end.

7 ADD(Kt1

Omni-swarm extracts map-based relative pose measurements by matching landmarks across keyframes, validating correspondences geometrically, and using accepted measurements for observability and drift correction.

  • Multi-drone map-based Localization Procedure:: Map-based localization builds sparse maps from landmarks and keyframes generated by local and remote measurements.The module uses these maps to extract relative localization measurements and reduce VIO drift.
  • Multi-drone map-based Localization Procedure:: Global descriptors retrieve similar keyframes, while landmark descriptors establish 2D–3D correspondences for relative pose extraction.Faiss retrieves nearest neighbors, with K set to 5 in practice.
  • Multi-drone map-based Localization Procedure:: Only successfully triangulated 3D feature points are broadcast to reduce communication among drones.The keyframe package includes global descriptors, landmarks, odometry, and extrinsic information.
  • Multi-drone map-based Localization Procedure:: Homography and PnP-RANSAC tests, followed by a gravity-consistency check, reject abnormal map-based correspondences.A measurement is retained when enough PnP inliers are found and the geometric test passes.
  • Relative pose extraction:: Map-based measurements provide sufficient relative-pose observability and eliminate accumulated VIO drift when drones revisit the same place at different times.The measurement can be modeled in any reference frame with Gaussian noise.

D. UWB Measurement

The back-end selects informative swarm keyframes, rejects inconsistent relative-pose measurements, and optimizes the resulting graph when observability requirements are satisfied.

  • D. UWB Measurement: Swarm frames combine VIO and UWB measurements; non-keyframes support forward propagation, while qualifying frames enter the graph.Keyframe selection is based on drone motion relative to the previous swarm keyframe or discovery of a new drone.
  • D. UWB Measurement: The graph retains at most mmax swarm keyframes, using random deletion to improve global consistency under computational limits.This replaces the sliding-window strategy used in the previous work.
  • Outlier Rejection: PCM provides unified outlier rejection for visual target and map-based relative-pose measurements.The module combines inter-drone and intra-drone consistency methods.
  • Outlier Rejection: Consistency graphs are updated incrementally, and each drone solves only PCM problems involving itself before sharing inliers.This design reduces computational power requirements.

2) Outlier Rejection for Distance Measurements:

Omni-swarm filters UWB distance outliers using elevation and residual tests, then constructs a robust factor graph for maximum a posteriori swarm-state estimation.

  • Outlier Rejection for Distance Measurements:: UWB measurements produce significant outliers at large relative elevation angles because drone airframes can occlude the signal.The system flags measurements using an elevation threshold set to 37° in practice.
  • Outlier Rejection for Distance Measurements:: A distance residual exceeding τd is rejected, with τd set to 0.3 m against an expected UWB error of no more than 10 cm.The threshold balances retaining correct measurements against allowing outliers to affect estimation.
  • Optimization Problem: The factor graph uses poses as variables and inlier measurements as factors for nonlinear least-squares MAP inference.Relative-pose factors include ego-motion, map-based, and visual detection measurements.
  • Optimization Problem: Huber loss reduces the effect of possible outlier factors across distance, map-based, and visual detection measurements.The relative-pose residual covers ego-motion, map-based, and visual detection factors.
  • Optimization Problem: Ceres Solver computes the optimization, while IMU-propagated VIO forward propagation supplies real-time swarm states at 100 Hz.The optimized poses are propagated to support high-frequency planner or controller inputs.

D. Observability Analysis

Omni-swarm combines complementary measurements to preserve relative-state observability across environments and provides multiple initialization routes based on available observability.

  • D. Observability Analysis: Multi-sensor fusion improves observability by combining measurements with different characteristics.Map-based localization and visual detection provide direct 6-DoF inter-drone observability in feature-rich scenarios.
  • D. Observability Analysis: Table I summarizes observability for typical measurement combinations, using T, F, and T/F to denote measurement availability effects.The combined measurements maintain observability across various complex environments.
  • D. Observability Analysis: UWB-odometry fusion supports feature-poor environments only when relative drone motion is present.Stationary drones or parallel flight do not satisfy this observability condition.
  • D. Observability Analysis: Omni-swarm tracks each drone’s observability independently and initializes or estimates only observable drones.This supports plug-and-play operation as drones become available.
  • D. Observability Analysis: Map-based localization can initialize an uninitialized drone from a single sufficient map-based measurement and the observing drone’s ego-motion.The resulting pose is inserted into the graph for subsequent estimation.

2) Anonymous Visual Detection Measurements for Initialization:

Omni-swarm initializes drone states from anonymous visual detections, UWB-odometry, and other synchronized measurements, while combining these inputs in real-time graph optimization. Its onboard implementation supports autonomous swarm experiments and map-based, visual, and UWB sensing.

  • Anonymous Visual Detection Measurements for Initialization: Anonymous visual detections provide 6-DoF information for state initialization, but their target IDs must be associated with uninitialized drone IDs.Omni-swarm uses a pruned depth-first search for this association.
  • Anonymous Visual Detection Measurements for Initialization: UWB-odometry initialization requires sufficient motion across multiple frames to achieve at least 3-DoF observability.The method solves an optimization problem and selects the lowest-cost result from three random initializations to reduce local-optimum risk.
  • Anonymous Visual Detection Measurements for Initialization: Omni-swarm checks each uninitialized drone’s observability before optimization and applies the corresponding available-measurement initialization once the condition is satisfied.Newly powered drones are added to available or uninitialized sets before this check.
  • Anonymous Visual Detection Measurements for Initialization: The system synchronizes measurements to UWB timestamps before fusion, using nearest timestamps for high-rate VIO and converted timestamps for visual and map-based measurements.UWB provides the swarm-wide time reference, while VIO’s 400 Hz output keeps conversion error small.
  • Anonymous Visual Detection Measurements for Initialization: Graph-based optimization runs at 1 Hz and is forward-propagated through VIO at 100 Hz for real-time onboard state estimation.The platform combines stereo fisheye cameras, UWB, and an Nvidia TX2 onboard computer; the software runs independently on each drone.

C. Redundant Computations

Omni-swarm distributes front-end computation across drones while redundantly solving back-end optimization on each drone. Experiments characterize computation growth, communication requirements, and network scalability as swarm size increases.

  • C. Redundant Computations: Distributed front-end computations include VIO, visual target tracking, and map-based localization, whereas graph-based optimization is redundantly computed on every drone.Redundant optimization improves robustness to temporary network or single-point failures and avoids distributed-optimization communication overhead.
  • C. Redundant Computations: Omni-swarm broadcasts odometry, UWB distances, keyframes, and PCM inliers over a one-hop wireless ad hoc network, with keyframes consuming the largest bandwidth.PCA compresses landmark descriptors from 256 to 64 dimensions and reduces communication by 534.5 kB per keyframe.
  • C. Redundant Computations: Packet loss does not change significantly with the tested number of drones, indicating network scalability, with an ideal laboratory-network swarm-size limit of 83.The laboratory environment provides 4.9 MB/s available bandwidth.
  • C. Redundant Computations: The front-end’s computation is largely independent of swarm size because most components run at fixed frequency, although visual pose estimation depends on detected drones in the field of view.The number of simultaneously visible drones is expected to remain limited in practice.
  • C. Redundant Computations: O(n) PCM outlier rejection scales linearly with swarm size, while graph-based optimization grows quadratically under the stated measurement-consistency assumption.The optimization problem’s scale grows quadratically with swarm scale.
  • C. Redundant Computations: Quadratic curves fit graph-optimization computation time in both simulated scenarios; PCM time is scale-independent in Scenario 1 but increases linearly in Scenario 2.Scenario 1 has only 2–4 neighbors per drone, while Scenario 2 provides relative measurements between all drones.

B. Features Verification

Feature-verification and dataset experiments assess fault tolerance, non-line-of-sight initialization, accuracy, and global consistency. Omni-swarm achieves the best overall ATE and RE among the compared methods, while ablations identify the roles of map localization, UWB, tracking, and outlier rejection.

  • B. Features Verification: When one drone automatically lands because of poor battery power, Omni-swarm’s estimation continues unaffected as another drone joins the experiment.The experiment demonstrates operation during a single-drone emergency and subsequent startup of a third drone.
  • B. Features Verification: With drones separated by a wall and outside mutual line of sight, Omni-swarm initializes when one drone visits the other’s starting place.The experiment disabled UWB because the modules could measure through the foam obstacle.
  • B. Features Verification: ATE measures global consistency, while RE measures relative state-estimation accuracy; reported averages use meters and degrees.ATE includes trajectory drift and relative-state error between trajectories.
  • B. Features Verification: The proposed method has the best ATE and RE overall on the recorded indoor datasets and also performs best on the simulated outdoor dataset.The comparison includes Parallel1, Parallel2, and RandFlight, plus ablations removing tracking, map localization, UWB, and outlier rejection.
  • B. Features Verification: Without Map-based retains good relative accuracy but its global consistency deteriorates to VIO-Only, demonstrating the importance of map-based localization for ATE.The ablation is especially relevant to feature-poor outdoor environments.
  • B. Features Verification: Without Outlier Rej. is two to four times less accurate than the proposed method on some datasets.The comparison attributes this difference to the outlier-rejection module and Huber norm.

3) Observablility analysis and initialization:

Omni-swarm is evaluated across observability, initialization, outdoor estimation, collision avoidance, real-time computation, and practical deployment scope. The experiments report accurate localization and initialization across tested environments, global consistency outdoors, and real-time operation, while identifying scalability, calibration, and communication-range limitations.

  • Observability analysis and initialization: Omni-swarm achieves accurate relative localization on all tested indoor and outdoor datasets, whereas competing methods fail or lose applicability in specific flight conditions.PGO is not applicable outdoors, and Xu2020 fails on the parallel-flight dataset because of a huge relative localization error.
  • Observability analysis and initialization: The proposed method initializes quickly before takeoff in every tested case and requires only one optimization.Xu2020 either fails to initialize correctly or requires post-takeoff flight, creating a safety concern.
  • Outdoor dataset evaluation: A 235-meter outdoor run produced 1.9 m of estimated-state drift, or 0.8% of trajectory length, while original VIO drift averaged 6.35 m.Map-based measurements remove VIO drift in the real-time estimate, while the final whole-trajectory estimate is smoother.
  • Inter-drone collision avoidance experiments: Omni-swarm supports independent real-time state estimation for inter-drone collision avoidance and planning in a four-drone experiment.The experiment validates practical operation in a real-world environment, although VIO can be more accurate on short flights.
  • Computation time: Omni-swarm achieves real-time performance on the onboard computer across the evaluated plug-and-play and collision-avoidance scenarios.The collision-avoidance scenario has longer back-end computation because it is more complex.
  • Conclusion and future work: The system reaches centimeter-level relative estimation accuracy while maintaining global consistency in extensive aerial-swarm flight experiments.The authors identify dependence on camera calibration, O(n^2) back-end scaling beyond 100 drones, and a 22.4 m communication range as future-work limitations.
Loading 2103.04131v5…