PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Embodied Air-Ground Cooperative Object Search: Benchmark, Dataset and Agentic Method

Boao Yu, Zimo Chen, Junreng Rao, Yue Hu, Zhengqiu Zhu, Yong Zhao, Rusheng Ju

arXiv:2609.08402v1cs.CVcs.AI

TL;DR

Urban air-ground object search requires UAVs and UGVs to coordinate aerial discovery with ground-level verification, but existing benchmarks do not directly evaluate this capability. The paper introduces AGOS-Bench, AGOS-Dataset, and a training-free AGOS-Agent that structures cooperation through search, handoff, and verification. Across nine VLM backbones, the agent improves success rate for eight and reduces decision steps for all nine, while important generalization and realism limitations remain.

  • Problem

    Existing benchmarks do not directly assess whether VLMs can coordinate UAV aerial exploration and UGV ground verification for urban object search.

  • Method

    AGOS-Agent uses role-conditioned VLMs, specialized tools, and a training-free search-handoff-verify protocol for UAV-UGV cooperation.

  • Results

    AGOS-Agent improves macro-averaged SR for eight of nine backbones and reduces DS for all nine, with mean changes of +22.5 percentage points in SR and −33.3 decision steps.

  • Takeaways & Limitations

    AGOS-Bench serves as a diagnostic testbed for heterogeneous embodied collaboration, while structured prompting alone does not solve the task.

  • Takeaways & Limitations

    Evaluation uses one test town with 210 episodes, static vehicle targets, idealized communication, known maps, high-level actions, and no simulator-to-real transfer.

Abstract

from arXiv · show

Air-Ground Object Search (AGOS) in urban environments is a challenging embodied task, which requires an Unmanned Aerial Vehicle (UAV) and an Unmanned Ground Vehicle (UGV) to jointly search for and verify a specified target vehicle from multi-view visual references. To study this underexplored problem, we introduce AGOS-Bench, the first dedicated benchmark for evaluating whether general-purpose Vision-Language Models (VLMs) can integrate aerial discoveries and ground-level verification through UAV-UGV cooperation. We further provide AGOS-Dataset as the companion resource of exemplary trajectories constructed by an automatic pipeline. It consists of 7.7k episodes for searching objects of diverse categories and attributes, spanning three difficulty levels. To address the AGOS task, we propose AGOS-Agent, a training-free and tool-augmented approach. The agentic method relieves VLMs from complex and dynamic coordination via a deliberate search-handoff-verify cooperation protocol, only demanding VLMs for scene understanding and decision-making. Extensive experiments on nine VLMs show that AGOS-Agent improves overall success rate for eight of the nine evaluated backbones while reducing decision steps for all nine. On the hard split, the SR and SPL of Gemini-3.6-Flash increase from 8.6% to 55.7% and from 7.6% to 44.0%, respectively.

Introduction

AGOS addresses the unassessed problem of coordinating UAV aerial discovery with UGV close-range verification for urban object search. The paper introduces a benchmark, dataset, and training-free agentic protocol to evaluate and support this cooperation.

  • Existing urban benchmarks do not directly assess whether VLMs can instantiate UAV-UGV systems for efficient open-space object search.
  • AGOS requires heterogeneous agents to coordinate autonomously under computation constraints while grounding targets across aerial and ground views.
  • AGOS-Dataset contains 7,700 episodes across five CARLA towns, three difficulty levels, disjoint train/test towns, and solvability guarantees.
  • AGOS-Agent uses role-conditioned VLMs, specialized tools, and a search-handoff-verify workflow while leaving geometric projection, path planning, and action validation to tools.
  • +22.5 percentage points in success rate and −33.3 decision steps are the mean paired changes across nine backbones, with SR improving for eight backbones and DS decreasing for all nine.
  • AGOS-Bench couples aerial exploration with ground verification in a closed loop and evaluates the stages with dedicated metrics.

Related Work

Related work spans embodied benchmarks, simulation platforms, and embodied navigation methods. AGOS extends this landscape toward heterogeneous air-ground cooperation and object-search evaluation.

  • Embodied benchmarks: Prior embodied benchmarks standardize observations, actions, and success metrics for indoor target search, while urban benchmarks address complementary search, question-answering, navigation, and spatial-reasoning tasks.
  • Embodied benchmarks: AGOS differs by evaluating cooperative UAV-UGV object search with confirmation or rejection rather than a question-answering answer or navigation endpoint.
  • Simulation platforms: Simulation platforms cover indoor navigation, outdoor urban scenes, aerial vision-language navigation, and heterogeneous air-ground embodied intelligence.
  • Embodied navigation methods: Embodied navigation methods increasingly combine pretrained vision-language models with world models, panoramic parsing, video planning, fine-tuning, and explicit spatial representations.

Task Description

AGOS defines instance-level urban object search in which one UAV and one UGV jointly search for a visually specified target and terminate after ground confirmation. The task emphasizes high-level collaborative capabilities while omitting several practical constraints.

  • Each episode assigns a target object through multi-view images and requires a UAV and UGV to search simultaneously within a predefined local region.
  • At each step, both agents use positions, multimodal RGB-depth observations, and inter-agent messages to choose actions through independent search policies.
  • The task ends when the UGV stops within 50 meters of the target and completes visual confirmation within its sight.
  • The setup evaluates wide-area search, multimodal grounding, and communication-based collaborative decision-making at a high level.
  • Localization errors, communication reliability and bandwidth, and other practical hard constraints are omitted for future work.

AGOS-Bench

AGOS-Bench evaluates cooperative urban object search in CARLA-Air, combining synchronized UAV and UGV observations, waypoint-based exploration, candidate handoff, and ground verification. Its dataset and metrics cover multiple difficulty levels and distinguish aerial search, candidate guidance, verification, navigation, and decision efficiency.

  • Benchmark setting: AGOS-Bench uses CARLA-Air to jointly simulate UAV flight and UGV driving with synchronized multimodal observations and urban-map access.The setup includes a nadir UAV camera, six UGV cameras, RGB, depth, and semantic maps, plus global localization, maps, and road networks.
  • Task protocol: The task constrains both platforms to discrete waypoints, with the UAV and UGV searching until the target is detected and the UGV confirms it visually.A UAV candidate region triggers notification for the UGV to travel there for verification.
  • Evaluation: The benchmark evaluates Success Rate, Oracle Success Rate, Success weighted by Path Length, and Navigation Error solely from UGV trajectories and verification outcomes.It additionally measures Aerial Search Efficiency, Aerial Candidate Guidance, Ground Verification Accuracy, and Decision Steps.
  • Dataset: The dataset contains 5,500 training episodes, 2,200 validation episodes, and 210 held-out test episodes on a separate unseen map.Expert trajectories are provided for training and validation, while the benchmark defines easy, mid, and hard configurations for vehicle-search tasks.
  • Automatic construction: Trajectory generation merges UAV and UGV waypoint graphs, solves a multiple traveling salesman formulation, and truncates trajectories at target discovery or handoff timing.When the UAV detects the target first, the solver appends a UGV path segment to the candidate region and records its arrival time.
  • Automatic construction: The automatic pipeline generates actions and observations at waypoints, samples images every 10 meters between nodes, and supports initial communication, inspection requests, confirmation, and hard-task rejection.Supplementary semantic images are also collected through CARLA’s sensor API.

AGOS-Agent

AGOS-Agent uses one general-purpose VLM in role-conditioned UAV and UGV policies, while a structured search–handoff–verify protocol and deterministic tools manage coordination and geometry. The UAV searches and annotates candidates, tools convert them into road-feasible targets, and the UGV verifies them.

  • Agent design: AGOS-Agent is training-free and uses the same VLM backbone as separate role-conditioned decision makers for the UAV and UGV.The roles differ in prompts, responsibilities, observation rendering, and legal action schemas.
  • Agent design: At each event step, the active role receives a role-specific observation and shared coordination state before producing a parsed legal action.The shared state includes candidate history, messages, agent locations, and the remaining decision-step budget, then updates after execution.
  • Coordination protocol: The standardized search–handoff–verify protocol assigns VLMs scene understanding and decisions while deterministic tools perform geometric projection and path planning.The design explicitly separates high-level reasoning from fundamental computations to handle dynamic coordination.
  • Search: During search, the UAV covers wide aerial areas and the UGV covers roads, after which the UAV annotates a single candidate when aerial evidence is sufficient.The two agents use complementary viewpoints: nadir RGB for the UAV and forward-facing driving views for the UGV.
  • Handoff: During handoff, Ego-to-Allo Projection maps the UAV’s annotated grid cell to the global map, and Road-Graph Path Planning selects a nearby serviceable approach node.The handoff uses one VLM decision and two deterministic tools to create a road-feasible UGV navigation target.
  • Verify: During verification, the UGV acquires a six-panel 360-degree observation and the VLM selects CONFIRM or REJECT, with rejection returning the workflow to aerial search.A false CONFIRM terminates the episode as failure.

Experiments

Across nine VLM backbones, AGOS-Agent improves success and reduces decision steps for nearly all models, while exposing bottlenecks in candidate guidance and ground verification. Stage-wise and trajectory analyses show how structured handoffs convert aerial observations into focused ground confirmation.

  • Overall performance: 55.7% SR and 44.0% SPL are achieved by Gemini-3.6-Flash with AGOS-Agent on the hard split, while the Human reference reaches 89.3% SR.The best VLM remains below the human reference on hard-split success.
  • Effect of the agentic protocol: +22.5 percentage points in SR and −33.3 decision steps are the mean paired changes across nine backbones.AGOS-Agent improves Overall SR on eight backbones and reduces DS on all nine.
  • Difficulty scaling: SR decreases from 48.8% on Easy to 26.2% on Mid and 23.2% on Hard across the nine AGOS-Agent runs.The larger Easy–Mid decline is consistent with wider-area exploration and longer-horizon memory demands.
  • Stage-wise bottlenecks: Gemini-3.6-Flash is balanced across stages with 68.5% ASE, 49.4% ACG, and 71.5% GVA, whereas Qwen3-VL-8B has 76.3% ASE but 33.4% ACG and 39.0% GVA.The contrast shows that informative aerial views are not reliably converted into precise handoffs and correct ground decisions.
  • Representative trajectories: AGOS-Agent succeeds in 29 decisions on a representative episode, whereas the baseline exhausts the 80-step budget without a structured candidate.The candidate is spatially close rather than pixel-perfect, yet it guides the UGV to an informative ground view.

Discussion and Limitations

The discussion attributes remaining failures to limited long-horizon search, imperfect cross-view grounding, and verification errors, while the evaluation scope constrains generalization and component-level conclusions. These limitations motivate uncertainty-aware verification and broader, more realistic testing.

  • Current VLM weaknesses: Performance falls when the search region expands, indicating limited long-horizon coverage memory and budget allocation.The paper identifies wider-area exploration as a distinct bottleneck.
  • Current VLM weaknesses: High ASE or OSR can coexist with low SR, so observation and approach do not guarantee successful cross-view grounding and verification.The paper also identifies inaccurate aerial beliefs being converted into confident false confirmations as a safety-relevant failure mode.
  • Decision budget and model scale: A decline in DS is beneficial only when SR also rises or is preserved.Decision efficiency must therefore be interpreted jointly with task success.
  • Limitations: The evaluation uses 210 episodes from one test town with static vehicles, idealized communication, a known map, discrete high-level actions, and no simulator-to-real transfer.Confirmation is scene-level rather than instance-ID or bounding-box prediction, and geographic and appearance generalization remain limited.
  • Limitations: The baseline-versus-full comparison evaluates the complete protocol rather than isolating individual tools, so component ablations and paired episode-level uncertainty estimates remain necessary.ACG is also specific to methods that construct structured candidate regions and cannot directly compare the baseline.

Conclusion

AGOS-Bench evaluates whether general-purpose VLMs can connect aerial search evidence with road-constrained ground verification through UAV–UGV cooperation. Across nine backbones, AGOS-Agent generally improves success and reduces decision effort, while exposing remaining bottlenecks.

  • AGOS-Agent improves overall SR for eight of nine VLM backbones and reduces decision steps for all nine.The best VLM still trails the Human reference by 19.3 percentage points.
  • Wide-area exploration, candidate grounding, and cautious final verification remain distinct bottlenecks in AGOS.These bottlenecks are identified through quantitative tables, success–efficiency analysis, and representative trajectories.
  • AGOS-Bench serves as a diagnostic testbed for reliable heterogeneous embodied collaboration rather than evidence that structured prompting alone solves the task.

A.1 Waypoint Construction and Feasible Options

Waypoint construction separates fixed map-level candidate locations from runtime-feasible destinations selected after environment filtering. This separation provides the waypoint basis for both UAV and UGV decisions.

  • Global waypoint sets are constructed and fixed before evaluation for each map.They define all UAV and UGV candidate locations available on that map.
  • At each decision step, the environment filters global waypoint sets into legal destination waypoints for the UAV or UGV.

A.1.1 Map-Level Waypoint Construction

Map-level waypoint assets use a filtered aerial grid and a topology-derived ground road graph. These assets are fixed and serialized for consistent episode-level evaluation.

  • UAV waypoint construction: The UAV waypoint space is a fixed two-dimensional grid above the CARLA map with 40 m spacing and 80 m flight altitude.The initial grid is defined over map bounds expanded by 50 m on each side.
  • UAV waypoint construction: A road-proximity filter removes aerial grid nodes that are too far from useful road context.The filter uses horizontal Euclidean distance to the UGV decision-anchor set.
  • UGV road graph: The UGV waypoint structure is generated from CARLA OpenDRIVE topology rather than a rectangular grid.A directed fine graph samples drivable lanes at approximately 2 m resolution, while a sparse decision graph provides high-level anchors.
  • UGV road graph: UGV decision waypoints combine approximately 10 m cruising anchors with topology-critical road anchors.Structural anchors include junction entrances and exits, forks, merges, and dead ends; fine-graph nodes between endpoints are retained for execution.
  • UGV road graph: The frozen road graph is serialized with its manifest and asset hash, while episodes store the expected graph identifier and hash for consistency checking.The episode waypoint field stores decision anchors, not every 2 m fine-graph node.

A.1.2 Runtime Feasible Waypoint Options

Runtime options constrain UAV movement to nearby active grid nodes and UGV movement to interpretable road anchors, while the evaluation metrics distinguish search, candidate quality, efficiency, and valid confirmation.

  • Runtime feasible options: The UGV receives high-level decision-graph targets, including forward anchors at approximately 10 m and 30 m of road-network distance.The selected anchor is reached by planning over the complete 2 m fine graph, keeping decisions compact while preserving road-valid motion.
  • Evaluation metrics: ASE measures how early the UAV makes the target oracle-visible within its search budget, not the accuracy of subsequently generated candidate regions.Higher ASE indicates earlier aerial observation.
  • Evaluation metrics: A valid confirmation requires a fresh STOP_OBSERVE context when enabled, target visibility, UGV distance no greater than 50 m, and clear line of sight when enabled.
  • Evaluation metrics: GVA is an episode-level conditional valid-confirmation rate and is undefined when GVA-Support equals zero.It must be interpreted with GVA-Support because a high value from few included episodes provides limited evidence.
  • Ablation interpretation: Candidate error must be interpreted jointly with SR and DS because low CErr can accompany delayed discovery and unsuccessful ground exploitation.Removing RGPP sharply lowers SR and SPL and increases decision steps, indicating that reachable ground execution is a major bottleneck.
  • Ablation interpretation: E2A can improve candidate placement when grid grounding is reliable but amplify incorrect grid-cell predictions under Qwen3-VL-4B.The no_e2a fallback can perform better in some easy episodes when the UAV footprint is already a reasonable target anchor.
  • Ablation interpretation: The ablation supports a coupled interpretation in which reliable confirmation depends on both candidate localization and an execution interface that enables efficient UGV approach and verification.Removing both E2A and RGPP shows that candidate generation and ground execution cannot be assessed independently.
Loading 2609.08402v1…