Source-linked AI summary
FlyPose: Towards Robust Human Pose Estimation From Aerial Views
Hassaan Farooq, Marvin Brenner, Peter Stütz
TL;DR
Aerial UAV operation requires reliable human pose perception despite steep viewpoints, low resolution, self-occlusion and onboard resource constraints. FlyPose addresses this with a lightweight top-down pipeline trained across aerial datasets, a manually annotated benchmark and edge deployment, reporting improved detection and pose estimation with real-flight validation.
Problem
Aerial human pose estimation remains difficult because steep viewing angles, low resolution and self-occlusion challenge accurate perception under real-time UAV constraints.
Method
FlyPose combines multi-dataset aerial training, a lightweight top-down detector-and-pose pipeline, the FlyPose-104 dataset and onboard edge deployment.
Results
FlyPose improves person detection across multiple aerial test sets and 2D pose estimation on UAV-Human, while supporting real-flight deployment.
Takeaways & Limitations
The deployed model provides a compact aerial pose-estimation backbone for UAV applications across varied viewpoints and embedded-system settings.
Takeaways & Limitations
Aerial detection still struggles with small objects, cluttered backgrounds, camouflage and ambiguous shadows, while facial keypoints remain especially difficult.
Abstract
from arXiv · showhide
Unmanned Aerial Vehicles (UAVs) are increasingly deployed in close proximity to humans for applications such as parcel delivery, traffic monitoring, disaster response and infrastructure inspections. Ensuring safe and reliable operation in these human-populated environments demands accurate perception of human poses and actions from an aerial viewpoint. This perspective challenges existing methods with low resolution, steep viewing angles and (self-)occlusion, especially if the application demands realtime feasibile models. We train and deploy FlyPose, a lightweight top-down human pose estimation pipeline for aerial imagery. Through multi-dataset training, we achieve an average improvement of 6.8 mAP in person detection across the test-sets of Manipal-UAV, VisDrone, HIT-UAV as well as our custom dataset. For 2D human pose estimation we report an improvement of 16.3 mAP on the challenging UAV-Human dataset. FlyPose runs with an inference latency of ~20 milliseconds including preprocessing on a Jetson Orin AGX Developer Kit and is deployed onboard a quadrotor UAV during flight experiments. We also publish FlyPose-104, a small but challenging aerial human pose estimation dataset, that includes manual annotations from difficult aerial perspectives: https://github.com/farooqhassaan/FlyPose.
1. Introduction
Aerial human pose estimation is difficult because steep viewpoints, self-occlusion, low resolution and onboard resource constraints obscure keypoints and limit usable computation. FlyPose addresses these challenges with a lightweight, multi-dataset pipeline, a new annotated benchmark, and real-flight deployment.
- Challenges: Steep sensor angles create foreshortening and frequent self-occlusion of human pose keypoints.Aerial views can obscure facial and leg keypoints.
- Challenges: Higher UAV altitudes reduce people’s image size, limiting the distance for stable pose prediction.Increased ground sampling distance leaves fewer pixels per person.
- Challenges: Onboard payload, dimension and power constraints restrict computation for higher-resolution imagery.These constraints limit available processing resources during flight.
- Contributions: FlyPose trains a top-down aerial pose pipeline across multiple datasets to handle top-down views and small scales.The pipeline is designed for deployment on edge devices.
- Contributions: FlyPose-104 provides manual bounding-box and pose annotations for challenging aerial imagery with frequent lower-body and facial self-occlusions.The dataset is presented as a challenging aerial pose-estimation test set.
2. Related Work
Existing aerial detection and pose methods lose accuracy under small scales, viewpoint shifts, occlusion and clustered imagery, while higher accuracy often requires costly models. The paper therefore targets a lightweight, generalizable pipeline for real-time UAV environments and validates it in flight.
- Existing approaches: Top-down pose methods detect person boxes first, then estimate keypoints on cropped person-specific regions.This focuses pose estimation on smaller patches with fewer background distractions than bottom-up methods.
- Data constraints: Aerial training data is fragmented across smaller application-specific datasets because low resolution, dense clusters and occlusion impede annotation.The cited applications include surveillance, traffic monitoring, search and rescue, cargo operations and remote sensing.
- Detection gap: 41 mAP on the VisDrone test set illustrates limited success for aerial detection, while higher small-object accuracy increases input size, complexity and latency.The passage motivates a more generalizable lightweight detector across UAV applications.
- Pose-estimation gap: Aerial viewpoint shifts and scale variation cause notable accuracy drops for lightweight pose architectures and conventional models.Top-down pose datasets are rarely explored outside controlled conditions, and existing aerial datasets remain limited in variety and scale.
- Paper direction: FlyPose adapts high-performing top-down models for diverse UAV environments under real-time constraints and tests the pipeline onboard a UAV.The work explicitly seeks to bridge the aerial top-view performance gap.
3. Methodology
FlyPose uses a top-down detector-then-pose pipeline for aerial video, selecting models through accuracy-latency benchmarking, multi-dataset finetuning and edge deployment. Its methodology includes separate detection and pose training plus the manually annotated FlyPose-104 dataset.
- System overview: FlyPose processes Full-HD multispectral video with a lightweight person detector followed by 2D keypoint estimation on detected regions.The system is designed for drone operation at altitudes up to 40 meters.
- System overview: The detector and pose estimator are trained separately before the combined pipeline is evaluated on edge hardware and deployed in flight.The system overview also connects aggregated pose information to downstream drone tasks.
- Training strategy: Model selection benchmarks architectures on aerial data for the accuracy-latency trade-off, followed by aerial finetuning and edge optimization.The strategy aims to improve results without significant computational overhead.
- Model design: RT-DETRv2 outputs person bounding boxes, while the pose module predicts heatmaps for 17 COCO body keypoints on cropped detections.The detector experiments with Normalized Wasserstein Distance Loss for more stable training on smaller aerial objects.
- Dataset: FlyPose-104 contains 104 manually annotated images and 193 people spanning varied backgrounds, 90-degree top-down views and heavy occlusion.Each person has a bounding box and 17 COCO pose keypoints.
- Preliminary testing: ViTPose-H achieved 67.52 mAP on UAV-Human among the tested pretrained pose models, motivating continued use of the ViTPose architecture.The preliminary comparison also considered latency and model size.
4. Experiments and Results
FlyPose is evaluated through separate aerial person-detection and pose-estimation experiments, followed by edge-device integration and real-flight deployment. Multi-dataset training improves detector generalization, while the selected pose model and complete pipeline meet onboard realtime constraints.
- Aerial Person Detection: 7.1 mAP higher average performance was achieved after initial VisDrone person-only finetuning across the used test sets.RT-DETRv2-S was initialized from COCO and Objects365 weights and finetuned for 60 epochs on VisDrone2019-DET.
- Aerial Person Detection: 6.78 mAP was gained on average after adding multiple aerial datasets, mainly due to thermal imagery improving HIT-UAV performance.The multi-dataset detector was evaluated across aerial test sets, with thermal data retained for nighttime applications.
- Aerial Pose Estimation: 73.18 mAP was achieved by ViTPose-H on UAV-Human, improving 16.3 mAP over the previously reported AlphaPose result of 56.9 mAP.ViTPose-H was the most accurate tested pose model, but its slower inference prevented deployment on the Jetson Orin AGX Developer Kit.
- Aerial Pose Estimation: 6.54 ms inference time made ViTPose-S the selected onboard pose model, while the RT-DETRv2-S and ViTPose-S pipeline reached 19.54 ms per image forward pass.The 19.54-millisecond figure excludes preprocessing and postprocessing; training included downscaling augmentation to mimic smaller or lower-resolution people.
- Deployment onboard the UAV: 20 ms from frame acquisition to pose prediction was reported during deployment, supporting 25fps realtime operation and leaving roughly another 20 ms for downstream tasks.The system was run on an embedded Jetson platform mounted with a multispectral gimbal camera on a commercial quadrotor during flight experiments.
5. Discussion
FlyPose improves aerial human detection and pose estimation through multi-dataset training, but performance remains constrained by small, ambiguous, occluded, and low-resolution subjects. The discussion also identifies dataset scarcity and modality-specific weaknesses as important limitations.
- Aerial imagery improves overhead predictions, but lightweight RT-DETRv2 detectors still struggle with small objects and cluttered backgrounds.Increasing the confidence threshold can also prevent distant people from being detected.
- RT-DETRv2-S generalizes across multiple test sets, viewpoints, and modalities with a small memory footprint and realtime feasibility on embedded systems.The authors note that its detection precision is below larger state-of-the-art models on individual datasets.
- Aerial HPE remains underexplored because pose-annotated datasets from aerial viewpoints are scarce, and FlyPose-104 exposes the difficulty faced by current models.The limited size and diversity of PoseFES did not significantly improve overall performance.
- Finetuned FlyPose better estimates self-occluded keypoints and performs relatively better on low-resolution patches than the COCO-pretrained ViTPose-S baseline.It still struggles with strong background camouflage and patches containing multiple nearby people.
- Thermal imagery produces less reliable poses than aerial RGB images, with more left-right keypoint swaps when low-resolution color information is unavailable.The swaps occur when the person’s facing direction is visually ambiguous.
- Facial keypoints receive lower average OKS scores than body keypoints on UAV-Human, while low-resolution heatmaps often fall below the 0.4 confidence threshold.The authors associate facial difficulty with occlusion and limited visibility.
6. Conclusion
The paper presents FlyPose as a lightweight aerial person detection and pose estimation model designed for embedded deployment. It reports competitive performance across aerial detection benchmarks, improved UAV-Human pose estimation, and validation through real flight experiments.
- FlyPose combines RT-DETRv2-S person detection with ViTPose-S pose estimation for aerial views.
- The person detector is trained on nine aerial detection datasets containing RGB and thermal imagery.
- FlyPose achieves competitive performance on the VisDrone, Manipal-UAV, and HIT-UAV test sets and improves UAV-Human pose estimation.
- The model maintains a small footprint suitable for embedded systems while delivering quality pose estimation across varied aerial viewpoints.
- Real flight experiments confirmed FlyPose results across multiple altitudes and aerial viewpoints.