PaperlayerView exact revision on arXiv

Source-linked AI summary

Borrowed Eyes: Markerless Nano-UAV Flight with an Active Quadruped Observer

Alejandro Lorite Mora, Dimitrios Arapis, Andrés Faíña

arXiv:2610.09967v1cs.RO

TL;DR

Nano-UAV payload limits constrain onboard localization, while fixed infrastructure is costly to deploy in industrial settings. A quadruped estimates the drone’s position from an arm camera and actively maintains observation; markerless estimates support autonomous flight with a median error of 59 mm.

  • Problem

    Nano-UAVs lack payload capacity for onboard localization sensors, while fixed external infrastructure is costly and impractical to install across industrial facilities.

    Source: §I · p. 1

  • Method

    A quadruped uses arm-camera segmentation and registered depth to estimate the drone in its map frame, while perception-aware NMPC adjusts body and arm to preserve observation.

    Source: B. Mask-Based Position Estimation · p. 2, E. Perception-Aware Active Observation · p. 4

  • Results

    58 to 60 mm median localization error against motion capture accompanied full autonomous flights using markerless estimates and inertial coasting through occlusions lasting up to 2.8 s.

    Source: C. Closed-Loop Flight on Markerless… · p. 6, §V · p. 8

  • Takeaways & Limitations

    A quadruped can provide a closed-loop markerless position stream accurate enough to pilot a nano-UAV without payload-intensive onboard fiducials or motion-capture flight estimates.

    Source: §V · p. 7

    The detector was trained on laboratory recordings with uniform backgrounds, confining experiments to that environment.

    Source: §V · p. 8

Abstract

from arXiv · show

Nano unmanned aerial vehicles (nano-UAVs) can navigate confined spaces that larger robots cannot, but their payload capacity severely limits the sensors and compute available for self-localization in global navigation satellite system (GNSS)-denied environments. We present a vision-based system that localizes a nano-UAV from a quadruped robot with an arm-mounted camera. The quadruped tracks the drone, estimates its position in its own coordinate system using segmentation masks and depth, and transmits that position over a real-time radio link. To ensure continuous tracking, we developed a perception-aware nonlinear Model Predictive Controller (NMPC) that dynamically adjusts the quadruped's body and arm to maximize the drone's visibility, treating observation reliability as a primary control objective. Relying solely on this external estimate, the nano-UAV executed predefined trajectories from takeoff to landing with a median 3D localization error of 59 mm. The system demonstrates robustness against short visual occlusions by smoothly transitioning to onboard inertial flight when line-of-sight is temporarily lost. Ultimately, this framework allows a quadruped to offload the localization burden of a nano-UAV, enabling inspection of complex spaces that neither robot could navigate alone.

I. INTRODUCTION

Nano-UAVs can reach confined inspection spaces, but payload limits constrain onboard localization and fixed infrastructure can be impractical. This work offloads markerless localization to a quadruped and actively maintains observation.

I. INTRODUCTION

  • Nano-UAVs can reach spaces inaccessible to larger robots, but their limited payload constrains onboard localization sensors and computation.Industrial inspection in narrow or elevated spaces is also time-consuming, costly, and hazardous when performed manually.Source: §I · p. 1
  • The quadruped estimates the drone’s position in its own global map and transmits it, providing a shared spatial frame for collaboration.The nano-UAV can scout confined spaces and report points of interest in a frame the ground robot can use.Source: §I · p. 1
  • The system replaces added drone sensing or environmental instrumentation with a mobile ground robot’s sensing and computational capacity.Fixed localization infrastructure is costly, inflexible, and often impractical across industrial facilities.Source: §I · p. 1
  • Perception-aware NMPC coordinates the quadruped’s body and arm to preserve line of sight, while onboard inertial flight bridges visual occlusions.The paper identifies reactive tracking losses during fast maneuvers or temporary occlusions as a limitation of prior work.Source: §I · pp. 1–2

II. RELATED WORK

Prior ground-aerial localization uses radio, external references, or magnetic beacons, while markerless visual observation has not served as the primary position source for closed-loop nano-UAV flight. Existing aerial-target and confined-space approaches also face target-scale or payload constraints.

Ground-Aerial Collaborative Localization

  • Ground-aerial localization has used UWB for decimeter-level localization, ground vehicles to refine onboard estimates, and legged robots as magnetic beacons with 8 to 11 cm tracking errors [6].Some approaches require dedicated onboard payload hardware, while others provide lower accuracy than close-range visual localization, [6].Source: §II · p. 2
  • Markerless inter-robot observation has refined multi-robot SLAM estimates [7], but had not been demonstrated as the primary external position source for closed-loop nano-UAV flight.The paper estimates only 3D position because the flight controller already estimates attitude and yaw, avoiding full 6D pose estimation [8], [9].Source: §II · p. 2

Vision-Based Localization of Small Aerial Targets

  • Prior aerial-target studies detected larger drones or relied on depth returns, whereas a bare nano-UAV presents a small surface area and very few depth points.The paper states it quantifies the relationship between target scale and depth returns up to 3.1 m.Source: §II · p. 2

Aerial Navigation Through Narrow Apertures

  • Collision-tolerant aperture platforms and caged inspection drones with onboard SLAM require structural protection or compute payloads beyond nano-UAV capacity.Nano-UAVs can access tighter spaces but cannot carry the heavy sensor packages assumed by those platforms.Source: §II · p. 2

III. METHOD

A quadruped-mounted RGB-D camera estimates the drone’s position and transmits validated map-frame estimates to a nano-UAV. The method composes the camera estimate with the quadruped’s localization and arm transforms.

A. System overview

  • A Spot quadruped’s arm-mounted RGB-D camera and onboard computer estimate at most one drone position per frame and send validated estimates by low-latency radio to a Crazyflie flying on its IMU.The onboard computer also handles quadruped control and arm articulation.Source: A. System overview · p. 2
  • Motion capture anchors the quadruped’s map and records evaluation ground truth, but never supplies state estimates to the drone’s flight controller.The quadruped’s onboard map drifted less than 1 cm at P95; motion capture was used to keep reported tracking performance independent of that self-localization error.Source: A. System overview · p. 2

B. Mask-Based Position Estimation

  • The map-frame drone position composes quadruped localization, odometry, arm kinematics, and the visually estimated camera-to-drone position.Because the drone’s IMU maintains attitude and yaw, vision estimates 3D position from a segmentation mask and registered depth; a fixed 4.5 cm offset moves near-face depth to the drone center.Source: B. Mask-Based Position Estimation · pp. 2–3

1) Sensing and Detection:

The sensing pipeline segments the drone in color images and pairs the mask with registered time-of-flight depth to estimate position. For this nano-UAV scale, reliable depth extraction is more challenging than 2D masking.

1) Sensing and Detection:

  • Reliable depth extraction is a greater challenge than 2D masking for a nano-UAV of this scale.The gripper camera streams registered color and time-of-flight depth at 640 × 480 and 60 Hz.Source: 1) Sensing and Detection · p. 3
  • The camera’s time-of-flight depth is registered to its color frame on the robot, supporting mask-based depth extraction.The deployed pipeline uses the registered depth stream alongside the detector’s color-frame segmentation.Source: 1) Sensing and Detection · p. 3
  • A YOLO11 instance-segmentation model segments the drone in each color frame in about 4.2 ms on the onboard computer.The model is exported to TensorRT at FP16.Source: 1) Sensing and Detection · p. 3

2) Auto-Labeled Training Data:

The paper trains its detector from automatically labeled in-domain imagery, then extracts depth with a sensor-specific strategy and integrity gates. Camera calibration and ground-truth comparisons distinguish raw accuracy from intrinsic estimator precision.

2) Auto-Labeled Training Data:

  • 7,306 positive frames and 1,099 drone-free negatives train the detector using motion-capture-prompted labels, composites, and no manual labels.Motion capture prompts the labeling pipeline only; the live detector never receives its data.Source: 2) Auto-Labeled Training Data · p. 3

3) Depth Extraction and Integrity Gates:

  • For the time-of-flight camera, nearest-supported-band aggregation places the drone within 250 mm in 93–96% of frames, versus 12–17% for median depth.The drone occupies only 6–14% of valid in-mask depths, causing median depth to lock onto the background.Source: 3) Depth Extraction and Integrity Gates · p. 3
  • Three gates reject unsupported clusters, implied jumps above 0.8 m/s, and inconsistent projected size; failed gates publish no position, so the drone coasts on its IMU.The stereo pipeline additionally uses a depth-completeness floor and background-annulus rejection; the time-of-flight camera does not need those gates.Source: 3) Depth Extraction and Integrity Gates · p. 3, Figure 2 · p. 3

C. Camera, Calibration, and Ground Truth

  • The gripper camera’s time-of-flight depth returns ranges on the dark airframe where stereo fails, while rigid alignment separates intrinsic precision from constant reference-chain offset.The comparison includes a bare airframe, a diagnostic matte panel, and an AprilTag fiducial baseline.Source: C. Camera, Calibration, and Ground Truth · p. 3

D. Single-Estimator Fusion, IMU Coasting, and Latency

The drone’s EKF fuses externally supplied position with onboard IMU attitude and yaw, coasting between updates. A perception-aware NMPC moves the quadruped’s body and arm to keep the drone observable while balancing viewing geometry and motion costs.

D. Single-Estimator Fusion, IMU Coasting, and Latency

  • The EKF fuses position updates with onboard IMU attitude and yaw, coasting between external updates as uncertainty grows.Each position estimate has 5 cm measurement noise; the approximately 180 ms pipeline latency leaves a position lag because flights fuse estimates on arrival without compensation.Source: D. Single-Estimator Fusion, IMU… · pp. 3–4

E. Perception-Aware Active Observation

  • The NMPC anticipates the drone’s known trajectory and minimizes observation-quality costs over a receding horizon to keep it near the optical center and within reliable depth range.It re-solves an eight-step horizon at 5 Hz and streams commands at 10 Hz; arm motion is weighted as cheaper than body motion.Source: E. Perception-Aware Active Observation · p. 4
  • The controller’s dominant penalty is optical-axis deviation, where estimator reliability degrades beyond approximately 30°; it also penalizes standoff distance from 1.0 m.The reactive baseline uses no prediction or cost trade-off and places the hand relative to the last received position.Source: E. Perception-Aware Active Observation · p. 4

F. Position-only control and the rate floor

Position-only control matched full-pose control on paired motion-capture trajectories, and the measured rate floor sets a target for the markerless feed. Live flights then evaluate localization accuracy, integrity gating, and comparison with fiducial tracking.

F. Position-only control and the rate floor

  • Position-only control matched full-pose control on every paired trajectory, while yaw drift was 9–10° per minute versus 0.8° under full-pose control.In a single-pass test, hover remained stable at 5 Hz and broke down at 1 Hz; latency, noise, and dropout tests remained preliminary.Source: F. Position-only control and the rate… · p. 4

IV. EXPERIMENTS

  • The evaluation covered eight closed-loop flights totaling 1,069 seconds, with an actively moving camera at 0.6–2.3 m stand-off.Offline hovering recordings assessed static estimation across 205 synchronized frames.Source: A. Markerless localization accuracy… · p. 4

A. Markerless localization accuracy and integrity gating

  • 49 mm median 3D error across 17,363 live positions, with a 119 mm P95; 99.4% of published positions stay within 300 mm of truth.The pipeline publishes at 20 Hz when the drone is visible, exceeding the measured 5 Hz stability threshold.Source: A. Markerless localization accuracy… · pp. 4–5
  • On stereo test figure-eight flights, markerless tracking had 122 mm median error, with 71 mm intrinsic precision after alignment versus 12–16 mm for fiducials.Alignment removed a 106 mm constant reference-chain offset; the markerless pipeline nevertheless achieved accuracy the authors considered sufficient for flight.Source: A. Markerless localization accuracy… · p. 5
  • A 1.8 g matte panel restored stereo depth but raised motor duty to 80.5%, causing saturation and emergency landing after 23 seconds.The bare airframe’s motor duty was 53–55%.Source: A. Markerless localization accuracy… · p. 5

B. Active Observation on the Quadruped

The perception-aware NMPC improved drone framing over reactive tracking, with an operational trade-off between tracking angle and the area swept by the quadruped. The reported comparison isolates controller tracking by using motion-capture ground truth.

B. Active Observation on the Quadruped

  • 0.54° pooled median tracking angle for deployed NMPC versus 7.4° for reactive tracking adjusted to equivalent 1.25 m standoff, a fourteen-fold framing improvement.The deployed tuning’s P95 angle was 2.7° over three flights; all controllers stayed within the 30° limit.Source: Table I · p. 5, B. Active Observation on the Quadruped · p. 5
  • Allowing body rotation yielded 0.70° tracking over 0.42 m², while sidestepping yielded 0.54° over a 1.72 m² footprint.The authors chose sidestepping because panning adds novel background clutter that degrades detection and stationary viewing improves depth precision.Source: B. Active Observation on the Quadruped · p. 5
  • Reactive flights had median off-axis angles of 10.5–10.9°, but their closer 0.86 m standoff geometrically amplifies angular error relative to NMPC’s 1.25 m.At equivalent 1.25 m standoff, the reactive lateral error corresponds to 7.4°.Source: B. Active Observation on the Quadruped · p. 5

C. Closed-Loop Flight on Markerless Vision Alone

Three 124 s flights completed ten-waypoint takeoff-to-landing trajectories using markerless estimates alone, with 58–60 mm median 3D error and no safety hand-back.

C. Closed-Loop Flight on Markerless Vision Alone

  • 58–60 mm 3D median error (P95 121–124 mm) was achieved across three ten-waypoint flights, each completed without a safety hand-back.Visual estimates remained available at 5 Hz or higher throughout the airborne duration.Source: C. Closed-Loop Flight on Markerless… · p. 6, Figure 5 · p. 6
  • 44 mm median lateral hover error was observed across three 120 s hovers, while visual estimates differed from motion capture by a 35–46 mm median offset.Hover drift remained within 24 mm laterally and 32 mm longitudinally.Source: C. Closed-Loop Flight on Markerless… · p. 6
  • 33 mm median motion-capture tracking error rose to 78, 88, and 109 mm for low, medium, and high markerless trajectories.Residuals peaked at direction reversals and settled within 45 mm on straight legs; the increase was attributed to 0.18 s pipeline latency, 56 mm median estimation noise, and larger high-elevation standoff degrading depth resolution.Source: C. Closed-Loop Flight on Markerless… · p. 6, Figure 5 · p. 6
  • Both 2.5 m attempts ended in onboard watchdog landing during ascent, while no safety hand-backs occurred across the tested configurations.All configurations used the deployed NMPC controller; the table is captioned as a clear-scene flown envelope.Source: C. Closed-Loop Flight on Markerless… · p. 6, Table II · p. 6

D. Loss of Line of Sight and Flown Envelope

The system recovered from human occlusions and completed a ladder-obstructed trajectory despite extended blindness, while the flown height envelope remained constrained by the arm’s elevation reach.

D. Loss of Line of Sight and Flown Envelope

  • During visual dropouts, the bridge withheld position updates and the drone coasted on its IMU, with safety hand-back at 4.0 s and watchdog landing at 4.5 s.The stress tests included human occlusions, a near-field ladder, and flights near the arm’s elevation limit.Source: D. Loss of Line of Sight and Flown… · p. 6
  • 0.22 m along-axis median drift contrasted with under 0.08 m cross-frame drift; 64% of blind-period drift was longitudinal.Longitudinal movement changes range rather than pixel location, while cross-frame motion affects tracking angle.Source: Figure 6 · p. 7, D. Loss of Line of Sight and Flown… · p. 7
  • 36 sustained human occlusions were all recovered, with first-sample reacquisition at 66 mm median error and no hand-back to motion capture.The longest blind interval was 2.78 s, below the 4.0 s safety threshold.Source: D. Loss of Line of Sight and Flown… · pp. 6–7
  • A 1.5 size-consistency threshold rejected 48% of false foreground depth locks while sacrificing 0.35% of valid frames.With that threshold, the drone completed the ladder-obstructed trajectory while blind for 48% of the flight and deviating at most 0.45 m from its planned path.Source: D. Loss of Line of Sight and Flown… · p. 7
  • Tracking reached 1.65 m flight height with over 98% availability; at 2.0 m, increasing standoff to 1.8 m restored availability to 77.7–84.5% and completed all waypoints.At 2.5 m, the arm’s kinematic reach was exceeded during ascent and the onboard watchdog landed the drone.Source: D. Loss of Line of Sight and Flown… · p. 7

V. DISCUSSION AND CONCLUSION

The quadruped’s markerless position stream enabled closed-loop nano-UAV flight, while outlier rejection and active observation supported operation through brief visual occlusions. The approach uses the quadruped’s integrated hand camera, but its tested range, environment, and evidence base bound the results.

V. DISCUSSION AND CONCLUSION

  • 49 mm 3D median localization error over 1,069 s of closed-loop flight; the markerless stream supported autonomous trajectories from takeoff to landing.The closed-loop flights had a 58 to 60 mm median localization error against motion capture.Source: §V · pp. 7–8
  • Depth-integrity gates rejected spurious readings as dropouts, eliminating all meter-scale background locks across 5,539 test frames; the pipeline provided 20 Hz updates against a 5 Hz stable-flight requirement.The authors report that the onboard IMU can coast through brief dropouts, while confidently false positions can destabilize flight control.Source: §V · p. 8
  • The quadruped’s integrated hand camera avoids extra sensor hardware on the ground robot, while the nano-UAV flies at its stock 37 g with no added payload.The hand camera combines high-resolution optical and time-of-flight depth sensors with factory kinematic calibration.Source: §V · p. 8
  • Experiments were confined to laboratory recordings with uniform backgrounds, and reliable tracking was reported between 0.6 and 2.3 m and up to 1.65 m flight height.Direct line of sight is required; a single viewpoint loses observability when the drone moves deep into cluttered structures.Source: §V · p. 8
  • The flight campaigns demonstrate practical viability rather than exhaustive statistical certification.The campaigns included three hovers, three full-loop autonomous flights, ten occlusion and clutter probes, twelve controller flights, and nine reference flights.Source: §V · p. 8
  • Future work targets broader robustness through more diverse training data and airframe modifications, joint path planning, and complementary sensing for cluttered structures.Proposed complementary modalities include radio-frequency ranging, optical beacons, and onboard relative tracking.Source: §V · p. 8
Loading 2610.09967v1…