PaperlayerView exact revision on arXiv

Source-linked AI summary

From Multi-Fisheye Sensing to Panoramic Perception: A Parallax-Aware Onboard Platform for Ultra-Low-Altitude UAVs

Dun Dai, Ze Lu, Cheng He, Yaowen Wang, Quan Quan

arXiv:2609.02319v1cs.RO

TL;DR

Ultra-low-altitude UAVs need surround vision that preserves context near obstacles while exposing raw fisheye views and calibration to downstream systems. This work builds an open, parallax-aware onboard platform for forming ERP panoramas, achieving improved geometric quality and sensor-rate embedded execution while supporting reusable perception interfaces.

  • Problem

    Ultra-low-altitude UAVs need panoramic context near obstacles, but existing systems do not combine synchronized raw-view access, parallax-aware ERP formation, and measured embedded sensor-rate execution.

  • Method

    The platform integrates four hardware-synchronized fisheye cameras, onboard computing, flight and navigation sensing, and a formation pipeline with overlap-wise projection-depth adaptation, controlled seams, photometric fusion, and cached remapping.

  • Results

    The accuracy-oriented profile reduces far-field P90 misalignment by 41.6% relative to Fixed Depth, while the deployed system sustains 19.99 frames/s at 13.29 W mean module-input power.

  • Takeaways & Limitations

    The results support an open sensing-to-perception interface that combines panoramic formation, embedded execution, and downstream vision access for panoramic robotics.

  • Takeaways & Limitations

    The formulation models one dominant depth per overlap and remains limited by mixed-depth content and real-time resource constraints.

Abstract

from arXiv · show

Ultra-low-altitude unmanned aerial vehicles (UAVs) require surround vision near buildings, vegetation, and other obstacles. We present a parallax-aware onboard platform that converts four synchronized fisheye streams into an open 1280x640 equirectangular panorama (ERP) interface. A purpose-built carbon-fiber airframe integrates the cameras, NVIDIA Jetson Orin NX, a flight controller, and a global navigation satellite system (GNSS) receiver. The formation pipeline selects projection depth per overlap and combines controlled seams and photometric fusion. Its accuracy profile adds content-adaptive seam search and a validation-gated residual mesh, whereas its deployed profile retains margin-gated Per-seam updates for sensor-rate operation. Evaluation uses more than 50,000 four-view groups from 18 field sequences. Relative to Fixed Depth, the accuracy profile reduces far-field P90 feature misalignment by 41.6%; the deployed Per-seam profile achieves the lowest aggregate geometric errors across held-out sites. Under a paced 20 Hz replay, the deployed profile sustains 19.99 frames/s at 13.29W mean module-input power. Eight-sector ERP sampling reaches 90.8% mean daytime visual-place-recognition Recall@5. Together, these results validate an integrated onboard panoramic-perception architecture that unifies parallax-aware formation, sensor-rate embedded execution, and reusable downstream vision interfaces for ultra-low-altitude UAVs. The project has been open-sourced at https://github.com/DUNDAI1998/parallax-aware-uav-panorama.

I. INTRODUCTION

The paper develops an open, parallax-aware onboard platform that forms four synchronized fisheye streams into ERP for surround-aware UAV perception. It combines adaptive panorama formation with embedded execution and evaluates geometry, photometry, runtime, and downstream interfaces.

  • Motivation: Ultra-low-altitude UAVs need panoramic vision because forward cameras can lose landmarks or objects near buildings, vegetation, and people.The ERP provides a continuous angular coordinate frame while retaining raw fisheye views for specialized processing.
  • Motivation: The remaining systems gap combines synchronized raw-view access, parallax-aware ERP formation, and measured embedded sensor-rate execution.Camera placement also determines overlap, baseline, and calibration, coupling mechanical integration to panorama formation.
  • Platform: The platform integrates four hardware-synchronized camera heads, a lightweight carbon-fiber airframe, Jetson Orin NX computing, flight control, and GNSS sensing.It exposes raw views, panorama, calibration metadata, and pose interfaces.
  • Formation pipeline: The pipeline adapts projection depth independently per adjacent-camera overlap, then applies dynamic seams, photometric correction, and two-band blending.Cached projection maps and multi-rate state updates reduce embedded per-frame workload.
  • Evaluation: The evaluation covers panorama quality, complete-path Jetson performance, frozen downstream perception, and localization on real UAV data.The final profile sustains 19.99 frames/s under a strict 20 Hz input at 13.29 W mean module-input power.

II. RELATED WORK

Prior work spans aerial panoramic sensing, multi-fisheye platforms, parallax-aware stitching, panoramic perception, and localization. This paper distinguishes its system by exposing raw views and calibration while measuring continuous onboard ERP formation and downstream interfaces.

  • Aerial panoramic sensing: Existing aerial systems use perspective cameras with inertial, GNSS, or LiDAR sensing, while related panoramic systems include annular-lens and six-fisheye platforms.The cited systems differ in platform type, sensing arrangement, and released interfaces.
  • Panorama formation: Prior parallax-aware stitching methods mainly target image pairs, whereas airborne use adds rig changes, exposure variation, temporal seam constraints, and bounded computation.These constraints motivate formation methods designed for multi-camera aerial deployment.
  • Downstream vision: Panoramic segmentation and depth methods address ERP distortion through domain adaptation, spherical geometry, or spherical attention, while localization work evaluates panoramic retrieval and related aerial interfaces.RflyPano and AirSim360 provide aerial panoramic simulation resources.
  • System distinction: The proposed system exposes four raw fisheye views, an ERP panorama, and timestamped navigation records through open data interfaces.Raw views preserve calibration access, while ERP supplies a regular interface to downstream algorithms.

A. Integrated sensing, computing, and interfaces

The integrated UAV combines a four-camera module, onboard computing, navigation sensing, and synchronized data handling in a compact platform. Its calibrated image path forms ERP onboard while navigation records are exported independently.

  • System integration: Fig. 2 combines the carbon-fiber airframe, camera heads, GNSS receiver, Jetson Orin NX, onboard ERP formation, and offline downstream evaluation.Navigation is exported as timestamped metadata alongside the formed panorama.
  • Computing: The Jetson Orin NX receives each synchronized four-view group and forms a 1280 × 640 ERP target in configured 25 W mode.The system target is one published panorama per 20 Hz input group without unbounded queue growth.
  • Synchronization: Each recorded visual group is stored as one synchronized four-view data unit, with navigation records associated by host timestamps.Flight-controller and GNSS records retain their native timestamps.
  • Dataset: The field record contains 18 complete sequences and more than 50,000 valid four-view groups across seven natural and built field sites.It includes daytime and night-time revisits and an additional near-field close-structure stress test.
  • Calibration: Kalibr-derived intrinsics and extrinsics bind camera order, geometry, and image-space refinement, and remain unchanged during evaluation.All camera transforms are expressed in a common rig frame using the left-front camera as reference.

C. Field data for evaluation

The evaluation uses synchronized four-view recordings spanning diverse sites and conditions, with sequence- and site-disjoint splits. The panorama pipeline separates per-frame processing from slower geometry, seam, and exposure updates.

  • C. Field data for evaluation: 18 complete sequences provide more than 50,000 valid four-view groups across seven natural and built field sites.The record includes teach–repeat flights and daytime and night-time revisits.
  • C. Field data for evaluation: Sequence- and site-disjoint samples prevent adjacent-frame leakage in panorama evaluation.The organized record includes synchronized fisheye views, corrected panoramas, calibration metadata, and navigation records when available.
  • C. Field data for evaluation: The visual core separates a per-frame CUDA path from slower geometry, seam, and exposure state updates.This design allows image quality to adapt without repeating every optimization on every frame.
  • C. Field data for evaluation: Each update tests feature correspondences against a projection-radius bank, uses photometric fallback when features are insufficient, and applies temporal hysteresis.Frames then undergo cached remapping, seam-local warping, dynamic seam estimation, exposure correction, and two-band blending.

A. Rig-aware projection and adaptive radius

The method models parallax by selecting a projection radius independently for each adjacent-camera overlap. Calibrated remapping and robust feature displacement statistics support depth selection with temporally gated updates and photometric fallback.

  • A. Rig-aware projection and adaptive radius: For each camera and candidate radius, cached remapping tables and validity masks encode calibrated fisheye projection into the ERP.The candidate radius bank is R = {1, 2, 4, 12, 30} m.
  • A. Rig-aware projection and adaptive radius: The projection-sphere radius acts as a discrete proxy for scene depth.Camera transforms and equidistant fisheye models map radius-dependent points into each camera image.
  • A. Rig-aware projection and adaptive radius: Each adjacent-camera overlap evaluates bidirectional SIFT matches after undistortion and ERP projection.The resulting Euclidean displacements use horizontal wrapping.
  • A. Rig-aware projection and adaptive radius: Q0.65 is the empirical 65th percentile of valid match displacements, measured in ERP pixels to limit isolated mismatches.Candidates require at least eight valid matches and a 0.18-pixel improvement.
  • A. Rig-aware projection and adaptive radius: A candidate update is accepted immediately at ΔSp ≥0.5 pixels; otherwise, two consecutive selections provide confirmation.Radius updates run every five frames, with coverage-constrained photometric scoring as fallback when features are insufficient.

B. Seam-controlled photometric fusion

The fusion stage confines adaptive warping and seam search to overlap bands, then combines exposure correction with separate low- and high-frequency blending. An accuracy profile can additionally retain a validated local residual mesh.

  • B. Seam-controlled photometric fusion: The 30 m layer supplies the base panorama, while each selected overlap layer replaces it within a 72-pixel half-band through a 12-pixel raised-cosine transition.Seam search is controlled separately by its own half-band.
  • B. Seam-controlled photometric fusion: A dynamic-programming seam minimizes a cost combining colour, smoothed colour, filtered colour, gradient, and edge terms.The image cost is p(y, x) = Dc + 0.75Ds + 0.35Df + 0.30Dg + 0.35E.
  • B. Seam-controlled photometric fusion: The deployed Per-seam variant fixes the seam-search half-band at 72 pixels, whereas Adaptive Seam tests 72- and 120-pixel bands.The wider path is accepted only when mean robust path cost improves by at least 0.01; displacement changes are limited temporally.
  • B. Seam-controlled photometric fusion: The full accuracy profile considers a residual-triggered local mesh only above 8-pixel overlap residuals with at least 15 mutual controls.Validation retains it only under specified median, P90, and seam-score conditions.
  • B. Seam-controlled photometric fusion: Exposure correction estimates per-camera gains from robust log-intensity differences and updates them every ten frames with exponential smoothing.Corrected warps are decomposed into low- and high-frequency components for subsequent blending.
  • B. Seam-controlled photometric fusion: Low- and high-frequency components use 16- and 3-pixel wrapped feathers, respectively.Weight fields are zero outside valid masks and normalized across cameras.
  • B. Seam-controlled photometric fusion: The fusion design smooths low-frequency colour changes without broadly averaging displaced foreground edges.This follows from separating the low- and high-frequency components during blending.

C. Panorama and navigation interfaces

Each synchronized capture exports raw fisheye views and a completed ERP with a shared image timestamp, while flight-controller and GNSS records remain separate navigation metadata. This preserves a reusable panorama interface without coupling image-quality and runtime paths to a state estimator.

  • C. Panorama and navigation interfaces: Each synchronized group exports four raw fisheye views and the completed ERP with a shared image timestamp.Flight-controller and GNSS records are retained independently as timestamped navigation metadata.
  • C. Panorama and navigation interfaces: The image-quality and embedded-runtime paths remain image-only, keeping the panorama interface reusable by downstream algorithms.The formation benchmark is not coupled to a particular state estimator.

D. Embedded implementation

The implementation separates cached per-frame CUDA processing from slower geometry, seam, and exposure updates, then validates quality and downstream interfaces across matched and held-out data. The deployed profile uses a complete embedded path while preserving reusable ERP outputs for detection and localization.

  • Embedded pipeline: The CUDA implementation caches maps, masks, and buffers, remaps each four-view group as an FP16 batch, and uses multi-rate state updates.Per-seam updates feature selection asynchronously every five frames and remaps the far layer over the full ERP.
  • Evaluation design: The evaluation compares panorama geometry, colour, complete-path execution, frozen object detection, and visual place recognition under matched protocols.Quality inputs are evaluated independently, while continuous clips test temporal switching and held-out clips test transfer.
  • Panorama quality: 41.6% lower far-field P90 feature misalignment is achieved by Ours relative to Fixed Depth.The residual mesh further reduces far-field median and P90 errors by 4.5% and 3.3% relative to Adaptive Seam.
  • Structural continuity: 50.8% of seam-local structures are preserved by Fast-margin, compared with 27.4% for Fixed Feather and 55.8% for Fixed Depth.Global Radius preserves 56.1%, while temporal-policy differences remain close to both endpoints.
  • Downstream interface: Eight-sector ERP sampling increases response coverage to 55.1%, compared with 39.4% for direct ERP inference and 46.4% for four sectors.The intervals quantify frame-level sampling uncertainty rather than retraining or ground-truth detection accuracy.
  • Held-out quality: Per-seam achieves the lowest three geometric errors and ties the best seam MAE in the continuous held-out quality comparison.Relative to Global Radius, its median, P90, and worst-overlap P90 errors decrease by 3.5%, 3.7%, and 2.2%.

B. Temporal-policy ablation

The temporal-policy ablation changes only acceptance timing while holding the panorama pipeline fixed. Fast-margin accelerates reliable depth-state updates without materially changing structural continuity relative to two-confirmation and immediate policies.

  • Acceptance policy: Fast-margin accepts a candidate after one observation only when improvement is at least 0.5 pixels; otherwise it requires two consecutive selections.Photometric fallback always retains two observations.
  • Experimental control: The ablation varies only temporal acceptance, keeping synchronized inputs, calibration, geometry, feature selection, seams, exposure, and fusion fixed.Embedded scheduling is evaluated separately.
  • Quality measures: The continuous quality table reports independent-AKAZE misalignment, worst-overlap P90, seam RGB error, and CIEDE2000 on held-out Lake and South-Square clips.Values are per-frame medians, all groups completed, and lower is better.

C. Sensor-rate performance on Jetson Orin NX

The Jetson benchmark replays complete four-view inputs at a strict 20 Hz gate and measures timing, drops, power, memory, and temperature. The deployed profile sustains sensor-rate output, while timely-output rate and latency expose a separate responsiveness trade-off.

  • Benchmark protocol: 20.0 ± the specified input-rate tolerance is required for accepted Jetson runs, with complete far-field and near-field sequences replayed without subsampling.All 36 profile and method runs passed the input gate, with a largest P95 arrival error of 0.235 ms.
  • Downstream validation: Figure 5 validates a frozen YOLOv10-N object-detection interface and a near-field line-structure continuity test as downstream checks.The detector uses matched post-takeoff far-field panoramas generated by Per-seam.
  • Throughput: 19.99 frames/s and 0.026% crosscondition timed drops are retained by both Per-seam and Global Radius.Their 50 ms timely-output rates are 71.04% and 99.92%, respectively.
  • Cost and latency: 13.29 W mean module-input power and 71.88 ms P95 latency are reported for Per-seam, versus 12.31 W and 29.40 ms for Global Radius.The configured 25 W mode is distinct from measured module consumption.

D. Visual place recognition for localization

Visual place recognition compares fisheye, rectified, and ERP representations under daytime and cross-time protocols, with sector density affecting retrieval outcomes. ERP 8V is strongest during daytime, while other conditions favor different sector densities, exposing a quality–efficiency trade-off.

  • VPR evaluates raw fisheye 4V, rectified 4V, ERP 4V, and ERP 8V with frozen CosPlace retrieval across daytime and cross-time protocols.Queries are sampled every two seconds within a 10 m GNSS radius, without temporal smoothing.
  • 90.8% Recall@5 is achieved by ERP 8V in daytime, alongside 74.9% Recall@1 and 17.2 m error.ERP 8V doubles descriptor calls relative to ERP 4V.
  • 23.8% South-Square day-to-night Recall@5 favors ERP 8V, whereas 77.8% South-Playground night-to-night Recall@5 favors ERP 4V.Preferred sector density is therefore condition- and budget-dependent.
  • Direct geometry and photometry remain the primary stitching evidence, while frozen detection and VPR provide complementary interface evidence rather than universal representation rankings.The conclusion frames sector density, temporal adaptation, and embedded cost as task- and budget-dependent choices.
  • The formulation remains limited by mixed-depth content and real-time resource constraints.These boundaries motivate richer local scene models while preserving open raw-view and calibration interfaces.
Loading 2609.02319v1…