PaperlayerView exact revision on arXiv

Source-linked AI summary

GAAT: Geometry-Aware Alignment Transformer for Multimodal UAV Perception

Jingpu Yang, Debin Tang, Yilin Sun, Fengxian Ji, Jiahua Zhu, Wenrui Ding, Yufeng Wang

arXiv:2608.27971v1cs.CV

TL;DR

UAV multimodal perception suffers from residual local misalignment that weakens token correspondence for contrastive learning and fusion. GAAT estimates reliability before interaction, uses it for sparse fusion and aligned supervision, and introduces paired-domain resources for transfer and acquisition-state evaluation. Across six downstream tasks, it reports state-of-the-art results on multiple UAV benchmarks and strong transfer, while its current validation centers on paired RGB–IR data and affine-supervised relations.

  • Problem

    UAV multimodal systems retain local spatial displacement after coarse alignment, challenging the correspondence assumptions used by contrastive supervision and dense fusion.

  • Method

    GAAT combines syncPATC reliability priors with geometry-guided sparse fusion, cross-granularity contrastive supervision, and UAVMeta and StateBench resources.

  • Results

    Across six primary downstream tasks, GAAT achieves state-of-the-art results on multiple UAV benchmarks and strong transfer across diverse perception settings.

  • Takeaways & Limitations

    StateBench extends evaluation from task transfer to camera reliability, observation scale, viewpoint stability, and flight maneuver complexity.

  • Takeaways & Limitations

    Current validation centers on paired RGB–IR data and affine-supervised relations, while extension to other modalities and more complex geometric variation remains future work.

Abstract

from arXiv · show

Unmanned aerial vehicle (UAV) multimodal perception integrates visible (RGB), infrared (IR), synthetic aperture radar (SAR), and depth sensors for scene understanding under diverse conditions. However, differences in optics, resolution, and mounting often limit practical systems to global or image-center alignment. After tokenization, parallax, platform motion, and lens distortion can shift corresponding patch centers across modalities, weakening the spatial correspondence assumed by dense contrastive learning and cross-modal fusion. We propose GAAT (Geometry-Aware Alignment Transformer), an alignment-first pretrained model that estimates local correspondence reliability before cross-modal interaction. GAAT introduces syncPATC, which learns patch-center consistency under synchronized view transformations without correspondence annotations. It emits geometric priors, including token and query confidence, query centers, and sub-token offsets, that identify reliable local anchors across residual misalignment. Guided by these priors, MG-Sparse-MMA performs query-mediated sparse fusion over top-K_s reliable regions, replacing dense all-patch interaction with geometry-calibrated local updates. RA-QCGCL aligns pretraining supervision with this sparse query bottleneck through reliable patch-to-patch, patch-to-query, and query-to-query contrastive branches. We introduce UAVMeta and StateBench, which provide four acquisition-state scores derived from platform telemetry and image statistics: camera reliability, observation scale, viewpoint stability, and flight maneuver complexity. Extensive experiments across six downstream tasks demonstrate consistently superior transfer performance, establishing GAAT as a state-of-the-art multimodal foundation model for UAV perception. StateBench further enables a systematic diagnosis of real-world acquisition conditions.

I. INTRODUCTION

UAV multimodal perception is limited by residual local misalignment and reliability-agnostic interaction after coarse alignment. GAAT addresses this with alignment-first reliability estimation, sparse geometry-guided fusion, matched contrastive supervision, and new UAV evaluation resources.

  • Motivation: Residual patch-center misalignment can corrupt same-index contrastive pairs and mix different physical regions during dense fusion.UAV frames also contain low-information patches, making all-patch supervision and interaction inefficient.
  • Motivation: GAAT targets local misregistration in coarsely aligned paired RGB–IR streams rather than regressing another RGB–IR warp.Its alignment-first pathway estimates interaction reliability before the modalities interact.
  • Resources: UAVMeta and StateBench support paired-domain pretraining, transfer evaluation, and acquisition-state encoding using four telemetry- and image-derived scores.UAVMeta includes paired RGB–IR imagery, annotations, and acquisition metadata.
  • Results: Extensive evaluation across six primary downstream tasks reports state-of-the-art results on multiple UAV benchmarks and strong transfer across diverse perception settings.The complete reliability, fusion, and supervision pathway has the strongest overall pattern in representative ablations.
  • Method: syncPATC learns transformation-consistent token and query confidence, query centers, and local offsets from synchronized synthetic transformations without correspondence labels.These priors provide the shared reliability signal for fusion and contrastive objectives.
  • Method: MG-Sparse-MMA and RA-QCGCL reuse reliability-ranked queries to restrict fusion and align patch- and query-level supervision with sparse interaction.The cross-modal interaction cost changes from O((HW)2) to O(KsLs), while encoder and within-modality attention costs remain unchanged.

C. Cross-Modal Registration and Representation Alignment

The section contrasts explicit registration with representation alignment and explains why UAV multimodal systems need local reliability modeling. It then describes UAVMeta’s paired data and StateBench’s acquisition-state metrics.

  • Cross-Modal Registration and Representation Alignment: Registration estimates image-space correspondences or transforms, whereas representation alignment constrains latent features without necessarily warping inputs.Existing UAV pipelines often use coarse image-center cropping or resizing instead of full pixel-level alignment.
  • Limitations of Registration: Hardware calibration and metadata projection require accurate sensor parameters and time synchronization that may be unavailable or unstable in large-scale UAV collection.Global affine or homography transforms also cannot fully handle parallax, oblique deformation, lens distortion, and timing-offset displacement.
  • Limitations of Registration: Appearance differences between RGB texture and color signals and IR temperature, material, or geometry responses make robust local matching difficult.This limits classical registration methods that depend on repeatable textures or structural keypoints.
  • Representation Alignment: syncPATC learns transformation-consistent confidence and local query offsets, while MG-Sparse-MMA and RA-QCGCL reuse these priors for fusion neighborhoods and contrastive relations.The shared reliability signal links alignment assumptions, supervision, and fusion.
  • Dataset Construction: UAVMeta contains 2,575 synchronized RGB–IR pairs across daytime and nighttime scenes, with acquisition-partitioned train/validation/test splits.The streams remain locally displaced by viewpoint, optics, and platform motion despite time synchronization.
  • UAV Acquisition-State Metrics: UAVMeta provides paired RGB–IR segmentation, detection, and classification annotations alongside CARS, OSGS, VSS, and FMCS acquisition-state scores.These resources support task evaluation together with acquisition-condition analysis.

D. StateBench: Unified Multimodal State Evaluation

GAAT’s pretraining and evaluation pathway combines modality-specific encoders, synchronized transformations, geometric priors, sparse fusion, and masked multimodal objectives. StateBench evaluates acquisition-state prediction from paired RGB–IR inputs using MSPA-4D and per-metric errors.

  • StateBench: Unified Multimodal State Evaluation: StateBench predicts four acquisition-state proxy scores from a single paired multimodal input using the MSPA-4D protocol.The scores cover camera reliability, observation scale, viewpoint stability, and flight maneuver complexity.
  • StateBench: Unified Multimodal State Evaluation: MSPA-4D uses tolerance-based, validity-masked, activity-weighted accuracy, with component weights and tolerances specified as hyperparameters.Activity weighting down-weights near-constant acquisition-state dimensions.
  • GAAT Pretraining Pathway: GAAT processes paired RGB–IR imagery through modality-specific HFE blocks, syncPATC reliability estimation, MG-Sparse-MMA fusion, and RA-QCGCL supervision.The pretraining objective additionally includes masked image modeling, knowledge distillation, and modality completion.
  • Synchronized View Construction: The synchronized-view pipeline jointly warps RGB and IR with a sampled affine transform, producing a known synthetic geometry and normalized view gap.The view gap combines normalized rotation, translation, scale, and shear components.
  • GAAT Pretraining Pathway: The full pretraining objective couples five terms, with alignment modules carrying the core geometry-aware design.HFE blocks, distillation, and modality-completion losses support the broader pretraining objective.

B. syncPATC: Synchronized Patch-Center Alignment

syncPATC learns local correspondence reliability from synchronized synthetic transformations rather than annotated cross-modal correspondences. Its priors rank reliable regions and calibrate query locations for subsequent sparse fusion.

  • Downstream Use: The learned confidence priors select top-K_s fusion regions, while sub-token offsets refine MG-Sparse-MMA sampling locations.Confidence priors used for contrastive learning are detached by default, while fusion priors may retain a lightweight gradient path.
  • Synchronized Transformation: syncPATC applies the same known affine transformation to RGB and IR while computing matching and inverse-warp cycle consistency within each modality.The resulting supervision concerns synchronized within-modality views, not direct RGB–IR registration offsets.
  • Query Centers and Correspondence: Attention-weighted query centers and affine-transform-guided correspondence define expected counterpart locations for local sampling.Validity indicators are obtained by warping the synthetic view mask to the same grid.
  • Reliability Learning: Correspondence and cycle-consistency losses pull queries toward matched tokens and enforce consistency after a round-trip transformation.Queries without valid neighbors are excluded from normalized correspondence and cycle losses.
  • Reliability Learning: A view-gap curriculum emphasizes reliable small-gap samples early and increases the weight of large-gap samples later.The schedule is described as an easy-to-hard weighting strategy.
  • Geometric Priors: The module emits token confidence, query confidence, query centers, and sub-token offsets after rescaling them to the fusion grid.These geometric priors are produced without manual correspondence labels.

C. MG-Sparse-MMA: Modality-Guided Sparse Multimodal Attention

MG-Sparse-MMA replaces dense all-token cross-modal interaction with reliability-ranked query-mediated updates. It uses geometric priors for selection and local sampling, then performs symmetric gated fusion with residual scatter-back.

  • Module Organization: MG-Sparse-MMA updates modality streams through a small set of reliable query slots instead of allowing every token to attend to every cross-modal token.The query bank extracts modality-conditioned queries from each modality stream.
  • Cross-Modal Query Bank: The shared query bank couples learned slot identity across modalities while retaining modality-conditioned query outputs.An optional UAV metadata prompt can be concatenated with the query bank.
  • Sparse Selection: The top K_s queries by reliability-weighted saliency restrict cross-modal interaction to a K_s×L_s neighborhood.This reduces interaction cost from O((HW)^2) to O(K_sL_s).
  • Geometry-Calibrated Local Sampling: The local sampling center averages modality attention centers and is refined by syncPATC center priors.Direction-specific learned offsets are combined with syncPATC offset priors for geometry-calibrated sampling.
  • Gated Fusion and Scatter: Bilinear local contexts are combined through direction-specific query projections and saliency-aware modality gates before residual write-back.RGB→IR and IR→RGB updates execute in parallel within each layer.
  • Role of syncPATC Priors: Token, query, center, and offset priors jointly reduce heavy attention to salient regions that are spatially mismatched across modalities.The design combines reliability-aware ranking, geometry-calibrated sampling, and symmetric bidirectional updates.

D. RA-QCGCL: Reliability-Aware Query-Guided Cross-Granularity Contrastive Learning

RA-QCGCL aligns contrastive supervision with GAAT’s sparse, reliability-ranked fusion by combining patch- and query-level relations selected from joint geometric and modality-specific evidence.

  • Joint reliability: Joint reliability combines syncPATC geometry with modality-specific image evidence to select credible tokens and queries for contrastive supervision.The reliability factor emphasizes tokens and queries whose correspondence is supported by image evidence and learned geometry.
  • Contrastive branches: P2P applies cross-modal InfoNCE to selected reliable patch pairs, retaining strict patch-index supervision only where local correspondence is credible.Selected token indices serve as positive labels, while pair-level registration quality reduces the contribution of lower-quality registrations.
  • Contrastive branches: P2Q assigns each selected patch to a semantic query slot and aligns it with the corresponding cross-modal query.This transfers supervision from reliable spatial patches to cross-modal semantic queries.
  • Contrastive branches: Q2Q aligns query slots across modalities and supplies an additional signal when patch-level reliability is weak.Because Q2Q operates on semantic query slots rather than token indices, it complements unreliable patch-level correspondence.
  • Objective: The three branches are jointly balanced in a reliability-weighted objective aligned with the sparse patch-and-query interaction structure.RA-QCGCL avoids reverting to dense patch-index assumptions after sparse query-mediated fusion has been adopted.

A. Experimental Settings

Experiments evaluate GAAT across external and in-domain UAV benchmarks using task-specific heads and modality configurations initialized from a common UAVMeta-pretrained encoder. Results cover segmentation, detection, and scene classification under their respective benchmark protocols.

  • Model configurations: All task variants use the common UAVMeta-pretrained dual Swin-V2-Base RGB–IR encoder with benchmark-specific heads and input branches.Heads include UPerNet, CMX/SegFormer-style decoding, DDQ-DETR, Global MLP, ChangeFormer, ChangerEx, and ByteTrack-style tracking.
  • Semantic segmentation: KUST4K segmentation reaches 82.14 mIoU and 92.41 mAcc, exceeding SGFNet by 1.07 mIoU points and DOFA by 2.93 mAcc points.GAAT(C) records the highest values in the table under the official evaluation protocol.
  • Semantic segmentation: UAVMeta segmentation reaches 67.79 ± 1.18 mIoU, improving over SegFormer-B5 by 4.27 points on the corresponding mean mIoU.The comparison uses an acquisition-disjoint split, and GAAT uses paired RGB–IR input while SegFormer-B5 is RGB-only.
  • Object detection: DroneVehicle detection reaches 56.59 mAP, 80.12 mAP50, and 67.16 mAP75, improving over DDQ-DETR by 6.49, 7.72, and 7.66 points.Gains occur across all three reported detection metrics.
  • Scene classification: Scene classification transfers to AID and RESISC45 with accuracies of 95.90/97.38 and 92.84/94.49 under the reported training ratios.SkySense or RVSA is stronger on several listed settings, so transfer is not uniformly best across every classification benchmark.
  • Scene classification: UAVMeta classification achieves 63.66 ± 1.78 mean accuracy and 51.97 ± 0.56 balanced accuracy, improving over EfficientNetV2-S by 0.23 and 0.72 points.GAAT uses both modalities, whereas the strongest baseline entries use a single modality.

E. Change Detection

GAAT is evaluated on change detection, tracking, novel-view synthesis, acquisition-state prediction, and component ablations. Results show strong transfer across spatially and temporally structured UAV tasks, with some benchmark-specific trade-offs.

  • Change detection: CDD change detection reaches 97.85 F1 and 95.79 IoU, within 0.03 and 0.06 points of ScratchFormer.On LEVIR-CD, GAAT achieves the highest listed F1 of 95.96 and IoU of 92.47.
  • Tracking: GAAT(B) has the highest pooled HOTA, IDF1, and MOTA on Drone 1 and the highest pooled HOTA and MOTA on Drone 2.CenterTrack retains the highest pooled Drone 2 IDF1 at 65.78 versus GAAT(B)’s 60.84.
  • Novel-view synthesis: GAAT+ThermalGS achieves 26.92 PSNR, 0.89 SSIM, and 0.11 LPIPS on TSDN, with the best SSIM and LPIPS.Thermal3D-GS and ThermalGS tie for the highest PSNR.
  • Acquisition-state prediction: StateBench reports an aggregate MSPA-4D score of 52.69 and the lowest FMCS MAE of 12.66 for GAAT.Other methods lead individual CARS, OSGS, and VSS MAE metrics.
  • Ablation study: The full GAAT configuration gives the strongest overall ablation pattern, while removing syncPATC, MG-Sparse-MMA, or RA-QCGCL weakens transfer on several benchmarks.The ablations preserve the remaining components and pretraining schedule unchanged.

J. Qualitative Analysis

Qualitative heatmaps show compact GAAT responses around task-relevant structures, consistent with reliability-ranked sparse routing and geometry-calibrated sampling. Cross-task results and acquisition-state benchmarks support geometry-aware alignment as a broad organizing principle, while current validation remains centered on paired RGB–IR data.

  • Heatmap comparison: Across five UAV scenes, GAAT responses remain compact around vehicles, road structures, and pedestrians, whereas MaRS activates broader background regions.The heatmaps use brighter colors for stronger responses and place both models on corresponding scene columns.
  • Heatmap interpretation: The compact responses follow top-K_s query routing, geometry-calibrated local sampling, and syncPATC offset refinement while retaining only the top quarter of patches.The maps are qualitative and align with ablation evidence that removing syncPATC or MG-Sparse-MMA lowers detection accuracy.
  • Cross-task impact: Across six primary downstream tasks, GAAT achieves state-of-the-art results on multiple benchmarks and is especially strong on segmentation, detection, change detection, and tracking.The paper relates this pattern to the spatial precision required by dense prediction and association tasks.
  • Acquisition-aware evaluation: UAVMeta and StateBench extend evaluation from scene content to camera reliability, observation scale, viewpoint stability, and maneuver complexity.StateBench is presented as a basis for condition-stratified evaluation, failure analysis, and future adaptation policies.
  • Scope and outlook: Current validation focuses on paired RGB–IR data, although the UAVMeta schema can accommodate SAR, depth, multispectral imagery, and other sensing streams.The paper identifies extension to these modalities and more complex geometric variation as a next step.

APPENDIX A ADDITIONAL EXPERIMENTAL RESULTS

Appendix results show GAAT transfers strongly across crowd counting, semantic segmentation, and object detection benchmarks, with leading performance on most reported metrics. Gains are especially consistent across additional UAV and visible–infrared datasets, though some modality-specific comparisons favor competing methods.

  • Crowd Counting: GAAT CountReg records the lowest listed MAE (9.40) and MSE (14.18) on VisDrone-CC.Lower values are better for both metrics under the reported evaluation protocol.
  • Semantic Segmentation: GAAT(U) leads UAVid with 82.31 mAcc and 69.98 mIoU, exceeding RingMo-Aerial(M) by 3.61 and 1.28 points.
  • Semantic Segmentation: GAAT(U) achieves the highest listed mIoU and aAcc on UDD5, with 77.17 mIoU and 90.31 aAcc.The margins over the strongest listed comparator are 1.71 and 0.68 points, respectively.
  • Semantic Segmentation: Across UDD6 and UAVM, GAAT(U) leads aAcc while GAAT(C) leads all three reported UAVM segmentation metrics.On UDD6, GAAT(U) has 89.95 aAcc but trails RingMo-Aerial(M) by 0.19 mIoU; on UAVM, GAAT(C) records 87.08 mIoU, 93.70 mPA, and 92.75 mF1.
  • Object Detection: GAAT(D) leads all reported metrics on HIT-UAV, LLVIP, UAVDT, and VisDrone-DET, with the largest listed UAVDT gains reaching 24.24 points in mAP75.On IndraEye, performance differs by modality: GAAT(D) remains below Oriented R-CNN for EO, while the supplied passage does not state the IR outcome.

D. UAVMeta Evaluation Protocol and Scope

UAVMeta evaluates scene classification, object detection, and semantic segmentation on synchronized RGB–IR pairs using an acquisition-disjoint split. Its qualitative figures complement dataset-level metrics, while change detection and tracking require external benchmarks because the current release lacks their annotations.

  • Evaluation Protocol: UAVMeta’s canonical acquisition-disjoint split contains 1,715, 572, and 288 synchronized RGB–IR pairs for training, validation, and testing.The release evaluates classification, detection, and segmentation, with fixed task-specific label protocols described in the passage.
  • Scope: UAVMeta 1.0.0-rc1 lacks change masks, temporal change labels, object identities, and trajectory annotations.Consequently, change detection and multi-object tracking are evaluated on external annotated benchmarks.
  • Qualitative Evaluation: UAVMeta’s qualitative figures show segmentation, detection, and change-detection inputs, predictions, reference annotations, and error maps.These examples complement the quantitative comparisons rather than replacing dataset-level metrics.
Loading 2608.27971v1…