PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Agile Vision-Based Multi-UAV Flight: Revisiting State Estimation

Michal Pliska, Matouš Vrba, Ondřej Víta, Martin Jiroušek, Viktor Walter, Martin Saska

arXiv:2609.39611v1cs.RO

TL;DR

Position-only estimation is widely used for target tracking, but multi-UAV systems using onboard relative localization lag the agility of individual UAVs. The paper compares position-only and pose-aware estimation and derives a linear thrust-direction-constraining KF. Pose-aware estimation reduces estimation error and delay, while the reported closed-loop validation remains in simulation.

Abstract

from arXiv · show

Agile multi-UAV flight requires accurate and low-latency onboard estimation of the kinematic states of neighboring UAVs for collision avoidance, motion coordination, etc. Most vision-based approaches rely on position-only measurements, inferring velocity and acceleration indirectly from displacement. We show that this introduces a fixed structural delay in the estimation of higher-order states, which limits the achievable agility. To address this, we propose to integrate tilt measurements, provided by a state-of-the-art visual detector, which inform about the thrust direction of co-planar multirotor UAVs. We benchmark four position-only and five pose-aware estimators, including a novel formulation of a linear thrust-constraining Kalman filter, on two real-world and one high-fidelity photorealistic simulated dataset over different levels of agility (3-21 m/s^2). In our setup, pose-aware estimation consistently reduces the average velocity and acceleration estimation errors by 40% and 57% across the three datasets with the proposed KF formulation outperforming the other estimators. Position-only filters exhibit a constant ~300 ms delay in acceleration step response independent of agility, whereas the tilt-constrained estimators operate near the physical response limit given by the camera frame-rate by observing the change in thrust direction before the displacement accumulates. In a closed-loop leader-follower simulated experiment with NMPC control, position-only estimation of the leader's state fails to facilitate stable hovering of the follower, while the proposed estimator enables tracking of lateral maneuvers exceeding 2g of acceleration.

I. INTRODUCTION

Multi-UAV systems using onboard relative localization have not matched the agility of individual UAVs, and prior evaluations have not systematically compared pose-aware and position-only estimation across agility levels. The paper situates that gap within onboard detection and target-state estimation methods.

I. INTRODUCTION

  • Multi-UAV systems using onboard relative localization lag individual-UAV agility, while existing studies typically test below routine single-UAV velocities and accelerations.The paper identifies a need for building blocks that address this research gap.Source: §I · p. 1

A. Onboard UAV detection

  • Monocular deep-learning detectors can provide full 6D target pose, while marker-based and spatial-sensor approaches offer alternatives with different equipment requirements.Marker-based detection can extract pose without domain-specific training; stereo or LiDAR provides relative 3D position but may require specialized, heavy, expensive sensors.Source: A. Onboard UAV detection · p. 2

B. UAV tracking and state estimation

  • Position-only tracking filters infer higher-order states from position, while prior pose-aware studies used limited dynamic excitation and did not report parameter-tuning methods.Position-only methods face a trade-off between lag at low process noise and jitter or instability at high process noise.Source: B. UAV tracking and state estimation · p. 2

C. Contributions

The paper benchmarks position-only and pose-aware estimation, introduces a linear KF formulation for directional acceleration constraints, and evaluates agile leader-follower flight in simulation. Its state model represents UAV acceleration through thrust, disturbance, and gravity components.

C. Contributions

  • The Degenerate Kalman Filter framework supports linear KF measurements defined by general dynamic affine subspaces, and the derived Z-KF uses this framework for pose-aware UAV state estimation.The paper reports that Z-KF outperforms other state-of-the-art methods.Source: C. Contributions · p. 2
  • The paper presents a systematic comparison with identical automated tuning, a photorealistic dataset spanning seven agility levels, and a simulated leader-follower experiment with a leader reaching 20 m s−2 acceleration.The experiment runs a full pipeline including control, onboard perception, state estimation, and target trajectory prediction.Source: C. Contributions · p. 2

III. STATE ESTIMATION

  • The UAV model decomposes total acceleration into thrust, external disturbance, and gravity components, with position, velocity, acceleration, and jerk represented in the world frame.The body z-axis is denoted b̂3, and thrust acceleration is defined from thrust force, mass, and orientation.Source: §III · p. 3, Figure 2 · p. 3

A. Point Mass-Model Kalman Filter

The baseline point-mass Kalman filters use position measurements with constant-velocity or constant-acceleration models. The paper evaluates analytical and block-diagonal process-noise covariance variants.

A. Point Mass-Model Kalman Filter

  • The baseline is a Kalman filter using only position measurements and a point-mass model, an approximation used when position alone is available.The paper describes this as representative of most state-of-the-art approaches.Source: A. Point Mass-Model Kalman Filter · p. 3
  • The point-mass filters use constant-velocity and constant-acceleration variants with corresponding state vectors and transition matrices.In both variants, the position measurement is modeled within the KF framework.Source: A. Point Mass-Model Kalman Filter · p. 3
  • Both variants are evaluated with analytical and block-diagonal process-noise covariance matrices.The block-diagonal variant assumes possible cross-correlation among position, velocity, and acceleration states, but not between their elements.Source: A. Point Mass-Model Kalman Filter · p. 3

1) Analytical covariance of the process noise:

The section describes two process-noise covariance variants and adapts the constant-acceleration Kalman filter to fuse tilt as a constraint on thrust direction.

1) Analytical covariance of the process noise:

2) Block-diagonal covariance of the process noise:

  • The block-diagonal covariance variant permits cross-correlation among position, velocity, and acceleration states, but uses more parameters and is more prone to tuning divergence.Its matrix is symmetric and consists of 3 × 3 diagonal sub-matrix blocks.Source: 2) Block-diagonal covariance of the… · p. 3

B. Z-Axis Measurement Kalman Filter (Z-KF)

  • The Z-KF represents tilt with the UAV body's upward-pointing z-axis and ignores heading because it does not contribute to acceleration.The acceleration magnitude is unknown, so each tilt measurement constrains acceleration to a line whose direction is the z-axis and whose offset is gravity plus drag.Source: B. Z-Axis Measurement Kalman Filter… · pp. 3–4

1) Degenerate Kalman Filter (DKF):

The DKF handles measurements that describe affine subspaces rather than conventional values, converting them into a form that a standard linear Kalman filter can fuse.

1) Degenerate Kalman Filter (DKF):

  • The DKF addresses degenerate measurements that represent affine subspaces of the state space, providing a recipe for a pseudo-linear Kalman filter.It defines a state projection similar to a Kalman filter with state equality constraints.Source: 1) Degenerate Kalman Filter (DKF) · p. 4
  • For an affine-subspace measurement, the measurement model combines a state mapping, a subspace basis with unknown scaling, and an offset.The model also includes measurement noise with covariance Z′.Source: 1) Degenerate Kalman Filter (DKF) · p. 4
  • Projecting with the null-space matrix removes the unknown subspace component, yielding a standard Kalman-filter measurement model that can be fused at each correction step.The projection uses N, the null-space matrix of W; the measurement variables W and o change at each correction.Source: 1) Degenerate Kalman Filter (DKF) · p. 4

2) Applying the DKF to the Z-Axis Measurement:

The Z-KF applies the DKF to fuse a tilt-derived acceleration-direction constraint with the CA-KF's position and acceleration measurements.

2) Applying the DKF to the Z-Axis Measurement:

  • The Z-KF uses the DKF to fuse a measurement model in which an unknown acceleration magnitude leaves acceleration constrained to a line along the measured tilt direction.The line is offset from the origin by gravity plus drag.Source: 2) Applying the DKF to the Z-Axis… · p. 4
  • The Z-KF configuration has nx = 9, nz = 5, nd = 6, and na = 1.The model also defines O as a 3 × 3 zero matrix.Source: 2) Applying the DKF to the Z-Axis… · p. 4
  • The resulting estimator is called the Z-KF, and the paper evaluates it with both analytical and block-diagonal process-noise matrices.Its update law is algebraically equivalent to the formulation in, while DKF provides a general framework beyond that application.Source: 2) Applying the DKF to the Z-Axis… · p. 4

IV. DATASET

The dataset section describes constraints in the existing laboratory benchmark, introduces a photorealistic simulation dataset with exact state ground truth, and reports detector pose errors across datasets.

IV. DATASET

  • The existing laboratory dataset uses a stationary camera and motion-capture pose ground truth, but lacks higher-order ground truth and has limited target motion, especially vertically.Higher-order states must be derived retrospectively, which may introduce inaccuracies.Source: §IV · p. 4
  • The released FlightForge dataset uses photorealistic simulation, moving-camera images, and target trajectories spanning 3–21 m s−2 agility and 1.5–8 m ranges.It provides exact bounding boxes, full 6D pose, velocity, and acceleration directly from the simulator, avoiding numerical differentiation of higher-order states.Source: §IV · p. 4

A. Detection and Pose Estimation

  • The resulting network’s median corner reprojection errors are 2.73 px on the Unreal dataset, 9.65 px on Phantom 4, and 5.36 px on Mavic 2; Unreal training used an independent set of approximately 13,000 images.Source: A. Detection and Pose Estimation · p. 5
  • 49 mm on Unreal, 106 mm on Phantom 4, and 43 mm on Mavic 2 are the median position errors; median orientation errors are 1.6°, 3.3°, and 1.7°, respectively.These poses provide the position and orientation measurements used by the evaluated estimators.Source: A. Detection and Pose Estimation · p. 5

B. Compared Methods

The comparisons span position-only CV and CA baselines and pose-aware filters, including the proposed Z-KF. Tuning and reporting use a held-out evaluation protocol and Mean Error Norm.

B. Compared Methods

  • Comparisons include the proposed Z-KF, baseline CV and CA KFs, Pose-KF, and an MEKF formulation modified for this estimation problem.Pose-KF incorporates tilt through the process-noise covariance; the MEKF uses either analytical or BDC process-noise covariance formulations.Source: B. Compared Methods · p. 5

C. Tuning and Evaluation Protocol

  • CMA-ES tuning uses dedicated dataset subsets, while reported results come from equal-sized disjoint held-out sets containing no YOLOv5-6D training or validation samples.Source: C. Tuning and Evaluation Protocol · p. 5
  • Mean Error Norm measures estimation error over valid samples in each test sequence, with d=9 for filters estimating position, velocity, and acceleration and d=6 for the others.The parameter vector is optimized in log10 space, with an additive λ=0.001 regularization term that helps stabilize convergence.Source: C. Tuning and Evaluation Protocol · p. 5
  • With d=6 tuning, estimator rankings were preserved and comparable metrics changed <1%, while including acceleration in supported filters' loss significantly improved acceleration estimation and prediction.The reported results use the median of ten independently tuned runs; BDC variants showed up to 175% difference in loss between worst and best runs.Source: C. Tuning and Evaluation Protocol · p. 5

D. Estimation Accuracy

Across datasets, thrust-direction information improves higher-order state estimates, with Z-KF+BDC strongest on Unreal. Pose-aware filters retain an advantage across agility levels, with especially clear gains in lateral estimation.

D. Estimation Accuracy

  • Position MEN stays about 0.10 m across methods; CV-KF has the highest velocity error, while its BDC variant reduces that error by 53% on average.Source: D. Estimation Accuracy · p. 5
  • On Unreal, Z-KF+BDC has the lowest velocity and acceleration errors, reducing MEN 42% and 58% versus the best position-only variant.Source: Table II · p. 6, D. Estimation Accuracy · p. 6
  • Pose-KF leads velocity MEN on Phantom 4 and Mavic 2 by 5% and 8% over Z-KF+BDC, but lacks acceleration states and assumes constant thrust magnitude.Source: D. Estimation Accuracy · p. 6

E. Performance over Agility

F. Estimation Latency

Position-only acceleration estimates incur a roughly fixed lag above physical onset, while the pose-aware estimate responds near the physical limit by using thrust-direction change before displacement accumulates.

F. Estimation Latency

  • In the 9 m s−2 step response, Z-KF+BDC responds near-instantly by sensing thrust-direction change before significant displacement accumulates.Source: F. Estimation Latency · p. 7
  • The experiment compares the best position-only and pose-aware filters on horizontal acceleration steps across seven agility levels, measuring delay by 63.2% rise time.Source: F. Estimation Latency · p. 7
  • CA-KF+BDC adds about 300 ms (7–8 frames) over physical onset at every agility level; Z-KF+BDC remains faster at all levels.Source: Table V · p. 7, F. Estimation Latency · p. 7

G. Closed-Loop Validation

In closed-loop simulation, Z-KF supports stable hovering and tracks aggressive lateral maneuvers, while CA-KF variants destabilize the follower. The discussion attributes estimation error and delay to position-only methods inferring higher-order states from accumulated displacement, and notes that closed-loop validation remains limited to simulation.

G. Closed-Loop Validation

  • The closed-loop comparison used a simulated leader-follower experiment with a 1.4 s NMPC prediction horizon and a full visual perception pipeline.The leader reached accelerations above 20 m s−2; estimator parameters were retuned using MEN error over the predicted trajectory.Source: G. Closed-Loop Validation · p. 7
  • Above 2 g, Z-KF maintained stable hovering and tracked lateral maneuvers for 20 s, while CA-KF variants destabilized the follower during hovering.The sequence ended when the leader left the camera’s field of view at the most aggressive waypoint, not because of estimator failure.Source: G. Closed-Loop Validation · p. 7, Figure 5 · p. 8

VI. DISCUSSION & CONCLUSION

  • Position-only estimation infers higher-order states from accumulated displacement, introducing estimation error and inherent lag that bound achievable agility in closed-loop control.The paper reports that incorporating thrust direction significantly reduces estimation error and delay.Source: §VI · p. 8
  • The closed-loop experiments were conducted only in simulation; real-world validation remains open.The authors also note that PnP-based pose extraction assumes known target dimensions, which may be limiting in some scenarios.Source: §VI · p. 8
  • The authors conclude that conventional relative-localization-based state estimation is insufficient for truly agile, communication-free multi-UAV flight.They provide a methodology intended to remove this limitation for further research.Source: §VI · p. 8
Loading 2609.39611v1…