Source-linked AI summary
Synthetic-to-Real ViT-Based Pose Estimation of a Noncooperative UAV
Krishnanujam Srinivas, Hanish Acharla, Brij Agrawal, Leonardo Herrera
TL;DR
Accurately labeled real-world UAV pose data is costly to collect, and synthetic-trained models must generalize across the synthetic-to-real gap. This work evaluates a DINOv2-based ViT trained on synthetic images and tested on real ones, using ambiguity-aware strategies and an α–β filter; filtering lowers mean angular error from 19.18° to 8.74° with inference time increasing from 13.25 ms to 13.42 ms.
Problem
Large-scale real-world UAV imagery with accurate pose labels is costly and time-consuming to collect, while synthetic-trained models face a synthetic-to-real domain gap.
Source: §1 · p. 1Method
A self-supervised DINOv2 ViT is trained exclusively on labeled synthetic imagery, evaluated on real imagery, and paired with ambiguity-aware strategies and an inference-time α–β filter.
Source: §1 · pp. 1–2Results
Mean angular error fell from 19.18° at 13.25 ms inference time before filtering to 8.74° at 13.42 ms after filtering.
Source: §6 · p. 11Takeaways & Limitations
The authors consider the post-filtering 8.74° mean angular error acceptable given identified contributing factors.
Source: §6 · p. 11The evaluation isolates inherent synthetic-to-real performance and does not use explicit domain-gap reduction techniques such as adversarial training or domain randomization.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Remote pose estimation of noncooperative Unmanned Aerial Vehicles (UAVs) from imagery is critical, as they cannot be influenced or instrumented in advance. Deep-learning-based approaches offer a promising solution; however, their development is constrained by the cost and difficulty of acquiring large-scale real-world datasets with accurate pose labels. Synthetic imagery provides an alternative, but models trained on synthetic data must overcome the synthetic-to-real domain gap to generalize to real-world imagery. This work investigates the inherent synthetic-to-real generalization capability of a Vision Transformer (ViT)-based model for monocular UAV pose estimation. The proposed approach employs a self-supervised DINOv2 backbone and is trained exclusively on labeled synthetic imagery while being evaluated on labeled real-world imagery. Pose ambiguity-aware strategies are incorporated during training and inference to address ambiguities arising from the projection of a three-dimensional target onto a two-dimensional image plane and from target symmetries. An $\alpha$-$\beta$ filter is further integrated during inference to improve pose estimations. To assess the model under operational requirements, it is evaluated in terms of Mean Angular Error (MAE) and inference time, both before and after filtering, using a real-world dataset containing 77,077 labeled UAV images. Before filtering, the model achieves an MAE of $19.18^{\circ}$ and an inference time of $13.25$ ms, whereas after filtering, these values are $8.74^{\circ}$ and $13.42$ ms, respectively.
1. Introduction
Remote pose estimation of noncooperative UAVs matters for operational scenarios that are difficult or impractical for human operators. This study tests synthetic-to-real generalization with a ViT-based model and incorporates pose-ambiguity strategies and inference filtering.
1. Introduction
- Large-scale real-world training data with accurate pose labels is costly and time-consuming to collect, while synthetic imagery offers labeled data under controlled variations.Synthetic-trained models must still overcome the domain gap to generalize to real imagery.Source: §1 · p. 1
- The study tests whether a ViT-based model can estimate UAV pose on real imagery after training exclusively on synthetic imagery.This setting is intended to assess inherent synthetic-to-real generalization without explicit domain-gap reduction techniques.Source: §1 · pp. 1–2
- The model uses ambiguity-aware strategies during training and inference to address pose ambiguity from 3D-to-2D projection and target symmetries.Distinct poses can produce nearly identical image appearances and large prediction errors.Source: §1 · p. 2
- An α–β filter is integrated at inference, and accuracy and inference time are evaluated both before and after filtering.The evaluation uses 77,077 labeled real-world images from a laboratory testbed dataset.Source: §1 · p. 2
2. Preliminaries
The preliminaries establish the rotation representation and error metric, a continuous six-dimensional rotation output, and a four-branch account of pose ambiguity resolved using radar-cue bits.
2.1. Rotation representation and error metric
- Orientations are represented by axis-angle vectors converted to rotation matrices, and prediction error is measured by geodesic distance on SO(3).The geodesic metric reflects angular displacement, with errors reported in degrees.Source: §2.1 · p. 2
2.2. Continuous six-dimensional rotation representation
- The network’s two three-dimensional output vectors are orthonormalized into a valid rotation matrix through the continuous six-dimensional representation.The resulting matrix has determinant +1 by construction, without post-hoc projection onto SO(3).Source: §2.2 · p. 2
2.3. Pose ambiguity and the four-branch symmetry group
- A fixed-wing UAV’s near-invariant silhouette yields four geometrically consistent orientations, whose distinct radar-cue sign-bit patterns identify the correct pose among the four.Figure 1 shows the four near-identical silhouettes and their differing cue-bit pairs.Source: §2.3 · p. 3, Figure 1 · p. 3
3. Method
The method combines a two-stage ViT pose decoder with synthetic appearance modeling and inference-time radar-cue correction. Its design addresses mirror-sensitive keypoints and ambiguity among pose orientations.
3.1. Model Architecture
- A DINOv2-backed ViT decoder first predicts three keypoints, then uses them with shared patch features to predict four more keypoints and aircraft orientation.The cascaded pipeline also estimates a left/right chirality logit and produces a raw per-frame orientation.Source: §3.1 · pp. 3–5
- The first-stage nose, tail, and fin define the body axis and fin side; stage 2 uses this cue to assign the mirror-sensitive wingtip and tail-surface pairs.The signed perpendicular fin offset distinguishes an aircraft from its mirror image, while the radar cue—not the fin-side cue—selects among the four candidate orientations.Source: §3.1 · p. 5
- Training scores orientation against the nearest of four mirror-related labels and mixes ground-truth and predicted cues so stage 2 can adapt to prediction noise.The chirality loss trains mirror-sensitive representations, but does not select the true candidate orientation; radar-cue correction handles that at inference.Source: §3.1 · pp. 5–6
3.2. Dataset Augmentation
- The adopted synthetic appearance model replaces binary silhouettes with spatially varying interiors, edge taper and halo, and low background noise calibrated to real-image statistics.Unlike flat-plate-plus-blur, it models correlated interior variation; the real imagery includes a dim halo and soft silhouette edges.Source: §3.2 · pp. 6–7
3.3. Inference-Time Radar-Cue Branch Correction
- At inference, two radar sign bits select the matching member of the four-pose orbit; simulated cues are not an oracle and may fail at zero crossings or when all branches are wrong.The reference bits expose only range-rate and elevation signs, and cannot recover a pose if the raw prediction is wrong on every branch.Source: §3.3 · p. 7
4. Experimental Setup
The protocol tests synthetic-to-real transfer by training on simulated imagery and evaluating on hardware-captured real SWIR images, with specified preprocessing, training, and geodesic-error reporting.
4.1. Dataset
- 77,077 labeled real SWIR images from a 3D-printed UAV mounted on a three-axis gimbal form the real evaluation dataset.Source: §4.1 · p. 7
4.2. Synthetic data and the synthetic-to-real protocol
- Training uses 100,000 synthetic silhouette images, while all 77,077 real SWIR images are reserved for evaluation.Source: §4.2 · p. 7
- Synthetic crops scale to the UAV bounding box, and all real and synthetic crops are resized to the fixed 336×336 network input.Source: §4.2 · p. 8
4.3. Implementation details
- Training uses a two-phase AdamW schedule that freezes then unfreezes the backbone, with the lowest synthetic-validation-loss checkpoint selected at epoch 102.Source: §4.3 · p. 8
- Table 2 compares three 200×200 training runs that differ only in how the synthetic silhouette is rendered.Source: Table 2 · p. 9
4.4. Evaluation protocol
- Evaluation reports mean and median geodesic error and the fraction of predictions exceeding 90° across all 77,077 real SWIR images.Source: §4.4 · p. 9
5. Results and Discussion
On 77,077 real UAV images, the single-frame model achieved 19.18° mean geodesic error, while temporal filtering removed the isolated near-180° error tail. The results also show gains from sensor-calibrated synthetic augmentation, radar-cue branch selection, and low-latency inference, alongside identifiable failure conditions.
5.1. Dataset augmentation ablation
- Calibration reduced mean error by a further 9.3° beyond generic augmentation, yielding a 44% total reduction from the raw-mask condition.The ablation attributes the gain to matching measured sensor statistics, rather than augmentation alone.Source: §5.1 · p. 9
5.2. Ambiguity resolution
- Radar-cue correction selected the correct mirror-related candidate more often, but 4.1% of predictions remained on a wrong candidate.The raw median error was 167.6° when the network selected the correct candidate only 18.1% of the time; the surviving errors form the near-180° tail.Source: §5.2 · p. 9, Table 3 · p. 9
5.3. Single-frame pose estimation
- 19.18° mean geodesic error, 10.58° median, and 38.42° RMSE were reported for single-frame estimates on 77,077 real images.The distribution also included 45.4% of predictions inside 10° and 4.1% beyond 90°; after filtering, the near-180° tail disappears and the main mode tightens around its median.Source: Table 4 · p. 9, §5.3 · p. 9, Figure 6 · p. 10
5.4. Sequential α-β refinement
- The α-β filter gates updates by residual size, coasting on kinematics when a measurement exceeds threshold τ.Its sequential refinement removes the isolated 180° tail shown in Figure 6.Source: §5.4 · p. 10, Figure 6 · p. 10
5.5. Inference latency
- 13.25±0.12 ms per forward pass was measured under fp16 with torch.compile, and the α-β filter added under 0.17 ms.The timing used one L40S at 200×200; the compiled forward pass corresponds to roughly 75 frames per second.Source: §5.5 · p. 11, Table 5 · p. 11
5.6. Qualitative failure analysis
- Catastrophic predictions occurred near degenerate viewing geometries and under oblique illumination that produced nonuniform reflectance.Near edge-on views, image parts collapse toward the same silhouette and the geometric cue becomes noise-dominated; broadside or moderate-angle views were more resolvable.Source: Figure 7 · p. 10, §5.6 · p. 11
6. Conclusion
The paper evaluates a DINOv2-based ViT trained only on labeled synthetic imagery against 77,077 labeled real UAV images, reporting an 8.74° mean angular error after filtering. It presents the model’s results as promising for real-world noncooperative UAV pose estimation and identifies future domain-gap reduction as a research direction.
6. Conclusion
- After filtering, the model achieved 8.74° mean angular error and 13.42 ms inference time on the real-world evaluation.Before filtering, the reported values were 19.18° and 13.25 ms, respectively.Source: §6 · p. 11
- The authors consider the filtered error acceptable given the synthetic-to-real domain gap, real-world pose misalignment, and inherent prediction error.They describe the results as demonstrating the model’s potential for real-world noncooperative UAV pose estimation.Source: §6 · p. 11
- Future work will investigate domain-gap reduction, including photogrammetric reconstruction of a 3D virtual model from a real UAV.The authors propose that this could generate labeled imagery retaining the real UAV’s visual characteristics.Source: §6 · p. 11