Source-linked AI summary
Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searching
Jaewoo Park, Minyoung Lee, Sukmin Seo, Moonbin Yim, Hyunwook Yoon, Dohoon Ryu, Daehee Kim, Myungseo Song, Jihyuk Byun, Seunggyu Chang, Taeho Kil, Jiseob Kim, Bado Lee, Geewook Kim
TL;DR
The paper asks whether MLLMs can act directly in a drone control loop when their action space is declared only in the prompt. It introduces a swappable-model agent and benchmark, finding that small models may navigate well but fail to follow the termination protocol, while deployment remains constrained by throughput sensitivity and task-comparison boundaries.
Problem
Prior systems increasingly narrow MLLMs’ decision-making, leaving unclear which parts of embodied drone control depend on the model itself.
Method
The paper builds DroneCATS-Agent with prompt-declared actions for pointing, searching, deliberating, and stopping, then holds the agent fixed while DroneCATS evaluates models across four drone-control capabilities.
Results
Small open models often reach the success radius more reliably than frontier models but fail to convert that navigation into success because they declare arrival prematurely or not at all.
Takeaways & Limitations
The central separation between models is action-protocol adherence rather than navigation, making correct self-termination a key requirement for deployable edge agents.
Takeaways & Limitations
Success is sensitive to serving throughput, especially for moving targets, and the commanding comparison is not a matched control because it also changes map and disambiguation demands.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) are strong perceivers of images and video. We ask how far that reach extends into acting: dropping an MLLM directly into a drone's control loop, with its entire action space declared solely in the prompt. Recent systems approach this setting but increasingly narrow the model's decision-making. We widen it back. We introduce DroneCATS-Agent, an architecture where the MLLM is a swappable component, and DroneCATS, a benchmark treating the model as the independent variable. Beyond merely flying toward a pixel, our agent entrusts the model to yaw and search, deliberate when unsure, and self-declare arrival---all without fine-tuning or function-calling schemas. Evaluating frontier and open models across four core capabilities---approaching a visible target, tracking a moving one, searching outside the initial view, and commanding a multi-drone fleet---reveals that even the simplest embodied settings are far from solved. Crucially, to identify what breaks first at the edge, our roster scales down to 2B parameters. The findings expose a stark paradox: it is not the flying that fails. Small open models often navigate into the success radius more reliably than frontier models, yet lose the episode by declaring arrival prematurely or not at all. Multi-drone commanding amplifies this divide, with small models failing by blindly copying a single coordinate across distinct views. Viewed as vision-language-action agents, the models' spatial perception holds up, but their action protocol does not. What separates a deployable edge model from a frontier model is not navigation, but the discipline to sustain a declared protocol and emit the correct terminating action. The open problem is closing this gap at onboard compute costs---yielding a fast model that plans persistently and knows exactly when it is done---and DroneCATS is built to measure that distance.
1 Introduction
The paper returns MLLMs to the drone control loop through a model-agnostic agent and evaluates their independent contribution across increasingly demanding embodied tasks. Its central finding is that navigation can succeed while protocol adherence—especially correctly declaring arrival—still fails.
- MLLM embodiment requires choosing physical actions from observations and responding to the observations those actions produce, with drone-control errors compounding over time.
- DroneCATS-Agent gives the model four prompt-declared actions—point, yaw, deliberate, and declare arrival—without fine-tuning or function-calling schemas.The architecture makes searching, deliberating, and stopping model decisions while keeping the surrounding controller fixed.
- DroneCATS varies target visibility and motion across approaching, tracking, and searching, then adds commanding one fleet of four drones toward look-alike candidates.The benchmark uses one unified success criterion anchored on the model’s arrival declaration, with 80 single-drone episodes and 20 commanding episodes.
- The simplest approaching cell remains unsolved: the best model succeeds in 13 of 20 episodes, while hiding the target initially reduces success to 8 episodes.
- Small open models often enter the success radius more reliably than frontier models but lose episodes by declaring arrival too early or not declaring it at all.The reported gap is therefore localized to protocol adherence rather than navigation.
2 Related Work
Prior UAV research commonly narrows MLLMs to planning, grounding, or structured subtasks, while existing benchmarks cover separate capabilities or use trained action mappings. DroneCATS instead holds the agent fixed, varies the model, evaluates one closed-loop protocol across tasks, and introduces multi-drone commanding.
- 2.1 Systems for Embodied UAV Agents: Prior systems progressively reduce the MLLM’s control role, from planning over detections or image candidates to emitting waypoints while separate controllers fly the drone.
- 2.1 Systems for Embodied UAV Agents: Existing system comparisons generally serve a fixed pipeline rather than isolating model capability; DroneCATS holds the agent fixed and varies the MLLM.
- 2.2 Benchmarks for Embodied UAV Agents: Existing benchmarks cover slices such as tracking, search, instruction following, or multiple-choice embodied reasoning, and several train the action mapping.
- 2.2 Benchmarks for Embodied UAV Agents: No prior benchmark evaluates approaching, tracking, and searching under one closed-loop protocol while treating the backbone as the primary axis and putting several drones under one agent.
- 2.3 Multi-Drone Agents: Prior multi-drone work mainly studies multi-view understanding or uses separate per-drone reasoning and control policies, whereas DroneCATS evaluates one MLLM controlling several drones from multi-view observations.
3 DroneCATS-Agent
DroneCATS-Agent gives an MLLM a prompt-declared, four-action interface for high-level drone control, while fixed geometric and flight controllers execute its choices. This expands beyond pointing by making search, deliberation, and termination model decisions.
- Action space: DroneCATS-Agent extends visual pointing with rotate, think, and finished actions, making search, deliberation, and arrival decisions explicit model outputs.The interface is declared in the prompt and requires no fine-tuning or function-calling schema.
- Execution: The go action couples image grounding with forward progress using a selected pixel and monocular depth estimate, without camera intrinsics supplied to the model.The resulting depth prediction relies on visual cues and learned scale priors before controller-side back-projection.
- Execution: Rotate enables out-of-view search, while think lets the drone hover during one extended-reasoning call when the scene is ambiguous.Rotation changes the view without translation, and extended reasoning is off by default because token latency consumes closed-loop time.
- Termination: Finished records the model’s arrival claim while the episode continues, allowing post hoc verification and supporting delegated multi-drone operation.The benchmark verifier decides whether the declaration was warranted rather than terminating on the model’s claim alone.
- Action space: The model receives egocentric RGB observations, instructions, and recent actions, then returns one JSON action per step.The four actions are go, rotate, think, and finished; go includes a point and depth, while the others use their specified control fields.
- Execution: A rule-based geometric controller converts go or rotate outputs into setpoints, leaving the model free of velocity and attitude control.For go, the controller back-projects the selected pixel and predicted depth into a body-frame displacement; rotate becomes a yaw-rate setpoint.
4 DroneCATS Benchmark
DroneCATS benchmarks a fixed model-agnostic agent across four single-drone task types and a four-drone commanding setting. A unified declaration-based criterion makes results comparable while testing whether models navigate, search, track, and terminate correctly.
- Task suite: DroneCATS varies target motion and initial visibility to form approaching, tracking, searching, and search-and-track task types.Approaching has a visible static target; search removes initial visibility, tracking adds motion, and search-and-track combines both.
- Success criterion: An episode succeeds when at least one arrival declaration occurs within δ = 5 m of the target in three dimensions while the target is visible.The same rule is applied across all four task cells, replacing task-specific scoring differences.
- Success criterion: The criterion counts qualifying declarations throughout the episode rather than judging only the final declaration.This handles temporary target nondetection and declarations made before the drone later enters the success radius while the scaffold continues flying.
- Success criterion: Passing through the goal region without declaring arrival is not success: 61 of 337 never-declaring failures had entered δ.Credit based only on trajectory proximity would have inflated the 182 successes by a third.
- Organisational settings: The benchmark evaluates one-drone and N-drone settings, with N=4 commands issued in one response from a context containing all four views.The model must preserve view-to-platform identity, so grounding a command in the wrong view is a distinct commanding failure.
- Organisational settings: The commanding suite uses approaching scenes with look-alike candidates distinguishable only by close-range inscriptions, requiring the fleet to divide candidates among platforms.The target is named by its inscription, and no drone can identify it from its starting pose alone.
- Maps and evaluation: The benchmark contains 100 AirSim episodes: 80 single-drone episodes across two maps and four task types, plus 20 commanding episodes.The single-drone suite uses ten episodes per map per cell; the commanding suite has ten episodes on each of two maps.
- Models and evaluation: The roster includes four frontier API models, four Qwen3.5 sizes, and Cosmos3-Edge-2B, with the small open-weight models included for deployment-oriented scaling analysis.Cosmos3-Edge-2B requires 10.5 GiB with a 32k context and decodes about 2 Hz on one H100, while onboard rates are expected to be lower.
5 Experiments
Across one-drone and fleet evaluations, performance is limited less by reaching targets than by sustaining the prompt-declared action protocol and terminating correctly. The benchmark also exposes sensitivity to target visibility, task type, stochasticity, and control-loop throughput.
- 5.1 Main Results: 65%: the best model succeeds when the target is initially visible, but no model exceeds 40% when the target is withheld from the first frame.The one-drone results cover 20 episodes per cell, with ten episodes on each map.
- 5.2 The Failure Is Declaration, Not Navigation: Qwen3.5-9B enters the success radius in 90% of episodes but converts only 35%, showing a declaration gap rather than a navigation gap.It declares at 0.63 of the start distance on average, while Qwen3.5-2B enters the radius in all four cells and converts none.
- 5.2 The Failure Is Declaration, Not Navigation: Qwen3.5-2B declares in 25% of episodes at 1.28 of the start distance, while Cosmos3-Edge-2B enters the radius in 25% and never declares.These opposite failure modes support the conclusion that protocol adherence, not perception or control alone, determines the small-model gap.
- 5.3 Commanding Several Drones: 70% of Qwen3.5-9B commanding cases emit an identical point for all four drones, compared with 1% for Gemini 3.7 Flash.Copying one answer across distinct views means at most one command can be grounded to its corresponding drone view.
- 5.3 Commanding Several Drones: 80%: Gemini 3.7 Flash succeeds in the four-drone commanding suite, whereas GPT-5 succeeds in 20%, despite approaching nearly as well in the one-drone setting.The commanding episodes add close-range candidate disambiguation and compare pooled fleet declarations, so the one-drone column is not a matched control.
- 5.4 Variance and Throughput Sensitivity: Three repeated Gemini 3.7 Flash flights produce 54.6±9.7% overall success, while per-cell variation of 9–13 points leaves the tier structure intact.The paper attributes this variation to episode-level stochasticity and cautions against over-reading neighboring model orderings within a tier.
6 Conclusion
DroneCATS shows that MLLMs can often navigate toward drone targets, but self-declared termination and fleet-level protocol adherence remain major weaknesses.
- 13 of 20 approaching episodes were completed by the best model, while hiding the target initially reduced performance to 8 episodes.
- Qwen3.5-9B entered the success radius in 90% of approaching episodes but succeeded in only 35%, indicating a protocol bottleneck rather than a navigation bottleneck.Qwen3.5-2B declared arrival at 1.28 times its start distance and never succeeded.
- The two best single-drone approachers completed 80% and 20% of commanding episodes, while small models repeated one point across four distinct views.
- Results are limited to simulation, and hierarchical commanding remains unevaluated.
Code
The authors provide the DroneCATS code and supporting materials needed to reproduce benchmark runs.
- The code will be available at https://github.com/naver-ai/DroneCATS.
- Appendix A provides the maps, prompts, and metric definitions needed to reproduce a run.
A Benchmark Details
The benchmark uses hand-authored, replayable episodes across residential and campus environments, with controlled visibility, motion, and multi-drone identification conditions.
- Episodes are authored in a browser editor that writes one replayable JSON specification per episode, including target, instruction, pose, distance, bearing, and initial visibility.
- The residential map uses repainted targets placed 16 m from each start, with controlled frame occupancy between 0.40% and 12%.
- Commanding comprises 20 episodes split across the residential map and AirSim Blocks, using look-alike candidates distinguished by inscriptions.
- Commanding starts place drones 12–32 m from named targets, while inscriptions are readable from 6 m or farther and do not require entering the 5 m success radius.
- Searching rotates the initial yaw by 90° on the residential map and by selected angles from ±90°, ±135°, or 180° on campus until no detections remain.
- Tracking targets move at 0.3 m/s residentially and 0.15 m/s on campus along three-dimensional interpolated paths, with a 5 m arrival radius.
A.2 Execution
Execution converts model-selected pixels, depths, and rotations into fixed rule-based drone commands under specified camera and simulator settings.
- The rule-based controller converts a selected pixel and depth into flight commands, rather than leaving execution constants to the action space.
- Back-projection uses a pinhole camera with focal length f = W/(2 tan(ϕ/2)) and image-centre principal point (W/2, H/2).
- Model coordinates normalized to 0–1000 become pixels (px, py) = (uW/1000, vH/1000), then map to body-frame displacement.
- The controller interprets depth along the optical axis and runs at ϕ = 90° with resolution 1280 × 720.
- A go action can yaw and then move up to 3 m for 2.5 s at no more than 1.2 m/s, while small vertical components are dropped.
- Rotation is executed as a 60°/s yaw-rate command, and 215 of 6401 rotations exceeded the prompted ±90° limit.
- The simulator uses AirSim 1.8.1 with SimpleFlight, a forward camera, and no PX4 SITL autopilot layer.
- Action history stores pixels while prompts use 0–1000 coordinates, causing round-trip rescaling by 1.28 horizontally and 0.72 vertically.
A.3 Serving and Decoding
The serving setup keeps the action space in a shared prompt while allowing provider-specific serialization and decoding. Each model receives the same high-level control interface, history, and image context, with thinking enabled only when requested.
- The system prompt is the only place where the action space is defined, and it is prepended before the image at every step.
- Open-weight models use enforced action-schema decoding, whereas API models decode freely; only two freely decoded replies failed to parse.The parsing result indicates that declaration failures are attributed to the prompt rather than output formatting.
- The prompt supplies coordinate conventions, depth guidance, search behavior, arrival criteria, and recent action history to support closed-loop control.The history is intended to help models detect circling or repeated mistakes, but the scaffold does not detect circling itself.
- Extended reasoning is disabled by default and enabled for the step after a THINK action, so deliberation incurs extra latency only when requested.A model that never emits THINK runs the episode at the cheaper setting.
- Models receive one egocentric image and must choose one JSON action, while provider adapters preserve the shared prompt, history, and image preprocessing.Provider-specific request envelopes differ, and no tools or function definitions are passed.
- DroneCATS exposes four model-selected actions: GO, ROTATE, THINK, and FINISHED.GO points toward visible targets; ROTATE searches out of frame; THINK pauses for deliberation; FINISHED declares arrival.
A.6 Metrics
The evaluation measures target distance, visibility, navigation error, and declaration-based success using simulator detections and recorded poses. These definitions distinguish reaching an object from reaching its center and make outcomes measurable even without a declaration.
- Distance is measured both to the target center and to the nearest point on its 3D bounding box.The two distances distinguish reaching a large object from reaching its middle.
- Visibility uses the simulator’s detection test, accounting for both the camera view frustum and occlusion.
- Navigation Error is measured at the decisive point: the first qualifying declaration, or the last action when none qualifies.This keeps the metric defined for episodes that never declare arrival.
- The runner records poses at 2 Hz and evaluates success within a 5 m radius of the target center under a 300 s episode cap.Declaration pose is tied to the frame from which the declaration was produced.
B Verifier Audit
The verifier audit shows that declaration-aware scoring materially changes conclusions compared with simpler alternatives. The adopted criterion scores declarations within the episode cap and preserves the distinction between arrival without declaration and declared success.
- Every verdict was re-derived offline from raw declaration and pose logs, matching the online runner across all 720 one-drone episode-runs.The audit also found that the three replaced criteria would change the paper’s conclusions.
- 15,714 declarations survived the episode cap, but 11,677 failed to qualify, so scoring only the first or last declaration would misclassify many outcomes.Scoring only the first declaration flips 95 successes to failure; scoring only the last flips 58.
- Requiring ten seconds of held visibility would fail 65 of 99 tracking-family successes despite declarations occurring inside the success radius with visible targets.The dwell rule is unsuitable when the target moves out of frame after a valid declaration.
- Crediting any trajectory entry into the 5 m radius would convert 61 of 337 never-declaring failures into apparent successes.That alternative would inflate the 182 scored successes by a third without requiring the model to claim arrival.
- The 300 s episode cap filters post-shutdown declarations that could otherwise turn failures into successes.In the present run, it removed 36 post-cap declarations across 12 episodes.
- The commanding figures are drawn from scored episodes, while Figure 1 comes from a recorded flight.
D Additional Results
Additional results show that map difficulty, fleet commanding, and diagnosed failure modes shape performance beyond aggregate success rates. Campus approaching is harder than residential approaching, while multi-drone failures can arise from copying one coordinate across distinct views.
- The best model reaches 8/10 on residential approaching but 5/10 on campus, and Gemini 3.7 Flash drops from 8/10 to 5/10.Residential targets are uniquely coloured vehicles, whereas campus referring expressions must disambiguate near-duplicate fixtures.
- Qwen3.5-4B repeats one point and depth across four views, so at most one of the four commands can be grounded.The views depict different scenes, making the repeated response a fleet-commanding failure.
- Figure 13 partitions episodes into success and six verdict-based failure modes, including declared, arrived, never grounded, oscillated, grounded, and timed out.Episodes that never moved are included in timed out but are identified as simulator faults and retried from the same start poses.
- Gemini 3.7 Flash assigns one car per drone and succeeds in 77 seconds, whereas Qwen3.5-27B sends the same point and depth to every drone.The comparison follows the same residential commanding episode and start grid.
- Across 80 one-drone episodes, frontier failures concentrate in mistimed declarations while small open models are distributed across every diagnosed mode.