PaperlayerView exact revision on arXiv

Source-linked AI summary

ESARBench: A Benchmark for Agentic UAV Embodied Search and Rescue

Daoxuan Zhang, Ping Chen, Jianyi Zhou, Shuo Yang

arXiv:2605.01371v1cs.RO

TL;DR

Existing UAV search-and-rescue research lacks a unified benchmark for evaluating embodied agents’ high-level reasoning and spatial memory. This paper introduces ESAR and ESARBench, finding that current agents face substantial challenges requiring aerial adaptation, spatial memory, and integrated decision-making.

  • Problem

    Traditional UAV SAR methods and fragmented benchmarks inadequately assess embodied agents’ high-level reasoning, spatial memory, and practical end-to-end capabilities.

  • Method

    The paper proposes ESAR and ESARBench, a high-fidelity UE5-AirSim platform with GIS-based environments, dynamic conditions, and event-driven rescue scenarios.

  • Results

    Baseline experiments show that direct transfer of ground policies is insufficient and that ESAR requires aerial-adapted perception, complex reasoning, spatial memory, and safety-aware search.

  • Takeaways & Limitations

    The benchmark frames autonomous aerial rescue as requiring integrated active perception, semantic reasoning, long-horizon planning, spatial memory, and robust decision-making.

  • Takeaways & Limitations

    All baselines remain far from solving ESAR, with the best SR and CDS reaching only 13.89 and 4.14, respectively.

Abstract

from arXiv · show

The rapid advancement of Multimodal Large Language Models (MLLMs) has empowered Unmanned Aerial Vehicle (UAV) with exceptional capabilities in spatial reasoning, semantic understanding, and complex decision-making, making them inherently suited for UAV Search and Rescue (SAR). However, existing UAV SAR research is dominated by traditional vision and path-planning methods and lacks a comprehensive and unified benchmark for embodied agents. To bridge this gap, we first propose the novel task of \textbf{Embodied Search and Rescue (ESAR)}, which requires aerial agents to autonomously explore complex environments, identify rescue clues, and reason about victim locations to execute informed decision-making. Additionally, we present \textbf{ESARBench}, the first comprehensive benchmark designed to evaluate MLLM-driven UAV agents in highly realistic SAR scenarios. Leveraging Unreal Engine 5 and AirSim, we construct four high-fidelity, large-scale open environments mapped directly from real-world Geographic Information System (GIS) data to ensure photorealistic landscapes. To rigorously simulate actual rescue operations, our benchmark incorporates dynamic variables including weather conditions, time of day, and stochastic clue placement. Furthermore, we create a dataset of 600 tasks modeled after real-world rescue cases and propose a robust set of evaluation metrics. We evaluate diverse baselines, ranging from traditional heuristics to advanced ground and aerial MLLM-based ObjectNav agents. Experimental results highlight the challenges in ESAR, revealing critical bottlenecks in spatial memory, aerial adaptation, and the trade-off between search efficiency and flight safety. We hope ESARBench serves as a valuable resource to advance research on Embodied Search and Rescue domain. Source code and project page: https://4amgodvzx.github.io/ESAR.github.io.

1 Introduction

The paper introduces Embodied Search and Rescue (ESAR) to address fragmented, instruction-following UAV research that lacks unified evaluation for autonomous SAR decision-making. It presents ESARBench, a high-fidelity, dynamic benchmark built from real-world terrain data with comprehensive evaluation and diverse agent baselines.

  • Motivation: Traditional UAV SAR uses decoupled perception and geometric planning, producing fragmented benchmarks that do not assess generalizable intelligence.These methods rely on narrow, predefined operational patterns and lack semantic reasoning.
  • Motivation: Existing embodied UAV benchmarks rely on fine-grained linguistic instructions, reducing agents to passive followers rather than active decision-makers.This differs from real-world SAR, where instructions are typically abstract and goal-oriented.
  • ESAR Task: ESAR requires agents to discover multimodal cues, reason about environmental semantics, and make autonomous decisions in complex 3D terrains.The task is designed for practical SAR scenarios and evaluates comprehensive embodied-agent capabilities.
  • ESARBench: ESARBench combines UE5 and AirSim with four large-scale GIS-based Chinese terrains spanning high mountains, desert, snowy peaks, and coast.The platform targets reduced visual sim-to-real gaps through high-fidelity rendering and accurate flight dynamics.
  • ESARBench: Dynamic rescue scenarios place mission-critical clues using event-driven temporal logic while varying weather and time of day.These conditions force agents to adapt to changing visibility and lighting in photorealistic environments.
  • Evaluation: The benchmark evaluates perception accuracy, reasoning capability, and mission efficiency across traditional exploration and ground and aerial MLLM-based agents.Results indicate that ground-policy transfer is insufficient and that aerial adaptation, complex reasoning, spatial memory, and flight-safety trade-offs remain challenging.

2 Related Work

Existing UAV search-and-rescue research relies heavily on traditional perception and geometric path planning, often requiring prior maps or probability models that are difficult to obtain in unpredictable emergencies. Although aerial navigation paradigms evaluate instruction grounding and visual navigation, they do not adequately assess the complex reasoning and spatial memory needed for end-to-end SAR missions.

  • UAV Search and Rescue: Traditional UAV SAR methods depend heavily on perception and geometric path planning, with many requiring pre-computed maps or predefined probability models.These prior-knowledge requirements are difficult to acquire under unpredictable real-world emergency conditions.
  • Aerial Navigation Paradigms: Aerial Vision Language Navigation extends ground-based VLN into 3D environments by evaluating navigation grounded in natural-language instructions and visual observations.The supplied passage identifies UAV-Flow and SPF as examples of aerial navigation work, but its description is truncated.
  • Limitations: Existing task paradigms fail to adequately assess complex reasoning and spatial memory, despite their importance for practical end-to-end SAR missions.This limitation motivates evaluation settings that target higher-level cognitive capabilities rather than navigation alone.

3 UAV-ESAR Simulator and ESARBench

Section 3 defines UAV-ESAR as an aerial embodied task requiring agents to explore 3D environments, discover semantic and spatial clues, and localize victims. It introduces a realistic simulator with diverse environments, sensors, dynamic conditions, hierarchical task generation, and metrics covering success, efficiency, clue discovery, and safety.

  • Task Definition: UAV-ESAR agents navigate complex 3D environments while using observations, internal state, prompts, and history to recognize semantic-spatial clues and locate victims.Unlike conventional navigation, the agent must explicitly output both flight actions and recognized clue information.
  • Task Definition: Victim localization succeeds when predicted coordinates are within error threshold E of the ground-truth coordinates, while clue reasoning is evaluated through Clue Recall Rate.Clue Recall Rate compares correctly predicted clues with the ground-truth clue set and contributes to the Clue Discovery Score.
  • UAV-ESAR Simulator: Four GIS-mapped Chinese rescue hotspots provide open-world environments spanning Alpine, Desert, Snowy Peak, and Coastal cliffs geomorphologies.The environments cover 2km x 2km, 2km x 2km, 3km x 3km, and 5km x 5km areas.
  • UAV-ESAR Simulator: The simulator places victims and 12 mission-critical clue types according to spatial and contextual patterns derived from historical real-world SAR incidents.Examples include tents, backpacks, discarded clothing, campfires, and signal flares.
  • UAV-ESAR Simulator: UAV-ESAR provides IMU, GPS, LiDAR, multi-view RGB, and depth sensors, with customizable conditions including 13 dynamically selectable weather types and varying time of day.Weather changes can induce corresponding physical changes in the simulated natural landscapes.
  • ESARBench Construction and Evaluation: ESARBench uses an Event-Snapshot-Task hierarchy and evaluates agents with SR, TSR, CDS, and RS, combining victim localization, temporal efficiency, clue discovery, and safe completion.RS uses Wsafe = 0.1, Wbase = 0.3, Wtime = 0.3, and Wclue = 0.3; CDS equally weights spatial localization and strict semantic matching.

4 Experiment

The experiment compares diverse ground and aerial exploration agents under a shared AirSim and perception setup, with APEX achieving the strongest overall performance. Results show that MLLM reasoning improves clue discovery, while aerial agents still face completion-awareness and efficiency limitations.

  • Baseline Setup: Baselines span basic exploration, non-MLLM and MLLM ground ObjectNav/VLN, and aerial VLN/ObjectNav methods under a shared AirSim and four-camera YOLO-World RGB-D setup.The evaluated methods include Random, FBE, Pure-MLLM, SemExp, VLFM, NavGPT, UniGoal, SPF, and APEX.
  • Overall Results: 13.89 SR, 4.14 CDS, and 13.45 RS make APEX the top baseline across victim search, clue discovery, and comprehensive rescue performance.SPF achieves the second-best RS of 13.12 and remains competitive across both victim search and clue discovery.
  • Clue Discovery: 3.48 average CDS for adapted MLLM baselines exceeds 2.70 for non-MLLM ObjectNav baselines, indicating the value of semantic reasoning for interpreting rescue clues.The results also suggest that MLLM reasoning is effective only when integrated into embodied search structures.
  • Limitations: 0.94 TSR for SPF and 0.87 TSR for APEX expose weaknesses in judging mission completion and improving efficiency while preserving broad search ability.Even the best SR and CDS remain only 13.89 and 4.14, showing that all baselines remain far from solving ESAR.

5 Conclusion

The paper introduces Embodied Search and Rescue (ESAR) as a bridge between embodied intelligence and critical real-world aerial missions. It also provides a high-fidelity UAV simulation platform and benchmark for quantitative evaluation, while baseline experiments demonstrate ESAR’s difficulty.

  • ESAR bridges embodied intelligence and critical real-world aerial missions.
  • The work develops the first high-fidelity simulation platform for UAV ESAR agents.
  • The benchmark quantitatively evaluates UAV ESAR agent performance.
  • Extensive baseline experiments underscore the profound difficulty of the ESAR task.

A Quantitative Difficulty Scoring

The benchmark quantifies embodied SAR task difficulty using victim-distance percentiles and environmental factors, then assigns tasks to four score-based difficulty levels. The scoring function combines distance, weather, illumination, victim count, and special-clue terms.

  • Quantitative Difficulty Scoring: Difficulty scoring uses the 25th, 50th (median), and 75th percentiles of average start-to-victim distance, incorporating weather, illumination, victim count, and special clues.These criteria form the quantitative difficulty scoring matrix for embodied SAR tasks.
  • Quantitative Difficulty Scoring: The task score is defined as S(T_i) = S_dist + S_weather + S_light + S_count + S_clue.The total aggregates distance, weather, illumination, victim-count, and clue components.
  • Quantitative Difficulty Scoring: Tasks are categorized as Simple (S ≤3), Medium (3 < S ≤5), Hard (5 < S ≤7), or Extreme (S > 7).The four levels are determined directly from the final score S(T_i).

B Calculation of the Success Rate

Success Rate is computed by optimally assigning predicted UAV coordinates to actual targets using spatial-distance costs and the Hungarian algorithm. A matched prediction counts as successful when its distance to the target is below threshold E.

  • Assignment formulation: Success Rate computation formulates associations between N predicted coordinates and M actual targets as a linear assignment problem.The formulation uses one-to-one prediction-target matching.
  • Optimal matching: An N × M cost matrix is constructed from spatial distances, and the Hungarian algorithm resolves ambiguous associations with optimal one-to-one matching.This addresses cases where multiple predicted coordinates cluster around a single actual target.
  • Success criterion: A prediction is successful when it is optimally matched to a target and lies within the predefined error threshold E.Both the matching condition and the distance condition are required.

C Details of the Four Reference Areas

The benchmark models four China-based reference areas spanning alpine meadows, desert and Gobi, snow-capped peaks, and coastal terrain, using realistic SAR-relevant conditions and mapped simulation regions. These environments include distinct hazards and geographic scales, from a 5 km × 5 km desert area to multiple 2 km × 2 km areas.

  • Reference-area selection: Four China-based regions represent alpine meadows, desert and Gobi, snow-capped peaks, and coastal areas for realistic SAR simulation.The selected regions have distinct topographies and high SAR-event incidence.
  • Map 1: Aotai Trail: Aotai Trail reconstructs a 2 km × 2 km Qinling Mountains area containing forests, alpine meadows, blockfields, and ridges under frequent extreme weather.At least 58 people have been reported missing or deceased there since 2012.
  • Map 2: Lop Nur: Lop Nur models a 5 km × 5 km desert-and-Gobi area centered on explorer Yu Chunshun’s tomb, with year-round aridity and frequent sandstorms.The region has a history of accidents involving scientific expeditions and explorations.
  • Map 3: K2: K2 incorporates a 2 km × 2 km summit area exposed to temperatures as low as -50°C, winds up to 50 m/s, and frequent avalanches.Since 1954, 92 fatalities have been recorded, with an approximately 20% fatality rate.
  • Map 4: Dapeng Peninsula: Dapeng Peninsula covers a 2 km × 2 km area around Wanglanggui, combining hilly terrain, forests, and coastal landscapes near Shenzhen.At least five fatalities have occurred there since 2025.

D Calculation of the Clue Discovery Score · E Justification of the Static Snapshot Formulation

The paper defines CDS through spatial localization and exact semantic matching, then justifies static temporal snapshots for ESAR evaluation using SAR behavior, kinematics, and negligible movement-induced misses. Experimental analysis also highlights a trade-off between exploration duration and flight safety.

  • D Calculation of the Clue Discovery Score: CDS evaluates reported clues at two accuracy tiers: spatial localization and exact semantic matching.A ground-truth clue is categorized according to whether it satisfies spatial and semantic conditions.
  • D Calculation of the Clue Discovery Score: Spatial localization counts a clue when reported and ground-truth coordinates are less than distance threshold E apart.This credits identification of suspicious regions even when semantic correctness is absent.
  • D Calculation of the Clue Discovery Score: Exact semantic matching requires distance < E and an LLM-identified valid match between reported text and the ground-truth clue name.The semantic criterion is therefore stricter than spatial localization alone.
  • E Justification of the Static Snapshot Formulation: Static temporal snapshots are presented as a justifiable ESAR approximation grounded in kinematics and SAR operational priors.The formulation is intended to represent dynamic missions while simplifying embodied-agent evaluation.
  • E Justification of the Static Snapshot Formulation: High-priority wilderness SAR targets are often incapacitated or remain in place, making displacement negligible during one search sortie.This behavioral prior supports treating victims as stationary within the evaluation horizon.
  • E Justification of the Static Snapshot Formulation: Because vd ≫vh, relative search velocity vrel ≈vd −vh cos θ is dominated by UAV velocity, making target motion marginal to search geometry.The kinematic argument treats victim movement as a secondary factor compared with UAV motion.
  • E Justification of the Static Snapshot Formulation: Movement-induced “perfect miss” evasion requires vh/vd ≥λRs/LB, but typical UAV speeds > 5 m/s and human walking speeds < 0.5 m/s violate this condition.The paper therefore characterizes the probability of movement-only misses as mathematically negligible in standard SAR operations.
  • E Justification of the Static Snapshot Formulation: Baseline results reveal a trade-off: stronger exploration ability often requires longer task time, while crash rates show safe long-horizon UAV operation remains challenging.Figure 5 compares crash rate, task time, and safe flight distance across baseline methods.

F More Results and Discussion

The paper presents ESARBench as an initial benchmark focused on life searching in open, non-disaster environments, while outlining a roadmap toward more complex scenarios, tasks, architectures, and modalities. Real-world deployment will require broader environmental volatility and thermal/audio sensing for hidden-victim detection.

  • Future Roadmap: Future ESAR scenarios progress from unconstrained open spaces and stable non-disaster events toward restricted enclosed areas and dynamic disaster zones.The roadmap scales deployment complexity across operational environments.
  • Future Roadmap: Agent capabilities are envisioned to expand from visual trace searching to risk assessment and targeted airdrops, alongside a shift from single-UAV autonomy to multi-UAV swarms.This roadmap jointly advances task formulation and system architecture.
  • Current Scope: ESARBench focuses on life searching in open, non-disaster environments to isolate multimodal perception, long-horizon reasoning, and spatial exploration challenges.The benchmark deliberately addresses the initial category of the broader ESAR roadmap.
  • Limitations and Extensions: Real-world deployment requires thermal/audio integration for hidden-victim detection and environmental volatility to test robustness.These extensions address limitations of the current benchmark scope.
  • Implementation: 140 hours of experimentation ran on a single A100 GPU using about 8G VRAM.The reported experiment consumed approximately 140 hours and 8G of GPU memory.

G Real-World Rescue Cases

The benchmark draws task-generation references from longitudinal real-world SAR incidents, discretizing each event into multiple static time snapshots. Four representative cases span hikers, explorers, and mountaineers facing injury, environmental hazards, disorientation, and suspected wildlife attack.

  • Event construction: Each complete, longitudinal SAR event unfolds over an extended period and is discretized into multiple static time snapshots for task generation.Figure 7 presents four representative event examples used as references.
  • Representative cases: Wu’s 2021 Aotai Trail incident involved injury, a missed rescue-team encounter, and death from hypothermia after he moved toward the summit.He was discovered on October 3, but the second hiker had left to seek assistance.
  • Representative cases: Yu’s 1996 Lop Nur traverse was disrupted by a severe sandstorm, leaving him unable to reach supply caches before he died from dehydration and heat exhaustion.He perished near a trail intersection several days later.
  • Representative cases: During a 1986 K2 descent, a severe blizzard disoriented two climbers, who ultimately died from a fall after three teammates reached safety.The team had successfully summited K2 before the storm struck during descent.
  • Representative cases: In 2025, Zhong explored the Dapeng Peninsula without professional equipment, showed suspected wildlife-attack telemetry, and was later found deceased with cause undetermined.He descended to the coastal area before being found.
Loading 2605.01371v1…