Source-linked AI summary
UAV-SwarmAgent: Scalable Multi-UAV Reconnaissance and Task Allocation via Code Generation and Vision-Language Reasoning
Oleg Sautenkov, Yasheerah Yaqoot, Muhammad Ahsan Mustafa, Grigorii Kalinin, Chih-Yung Wen, Dzmitry Tsetserukou
TL;DR
UAV reconnaissance can shorten missions but requires an operator to manage waypoints and monitor multiple video streams, while the duration of language-commanded multi-UAV missions remains difficult to anticipate. UAV-SwarmAgent turns one natural-language instruction into routes for multiple UAVs and has onboard agents verify incidents. In simulation, it located incidents in 244 of 255 missions, and its measured flight-time scaling makes the benefit of adding drones predictable.
Problem
Multi-UAV reconnaissance adds waypoint-planning and video-monitoring work for the operator, and evidence is missing on turning one instruction into a multi-drone mission with an anticipatable duration.
Source: §1 · p. 1Method
A ground-station agent uses code to ground inspection targets and allocate routes, then delegates camera-frame incident verification to one agent per UAV.
Source: §1 · p. 1, §3.1 · p. 2, §3.3 · p. 2Results
Takeaways & Limitations
The operator’s input does not grow with the number of drones, while the waypoint overlay before dispatch and returned frames afterward are places to catch errors.
Source: §6 · p. 4The study used synthetic incidents and a kinematic flight model without wind, batteries, or obstacles, and involved no human participants.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Supervising a multi-UAV reconnaissance mission usually means plotting waypoints and watching every drone. We present UAV SwarmAgent, a hierarchical multi-agent system in which an operator states the mission in one natural-language sentence and a ground-station agent writes and executes code that turns it into per-UAV routes. The agent grounds inspection targets in an aerial map with a vision-language model, splits them among N UAVs by balancing route length, and dispatches worker agents that verify incidents with a compact onboard decision model. In simulation over 4 incident types and 3 foundation models, the swarm located the incident in 244 of 255 missions (95.7%). A paired scaling study with one to 5 UAVs shows that flight time follows an Amdahl-type law. The second UAV delivers half of the attainable saving and the speedup is bounded at 1.52x. We discuss what this predictability, and the false alarms we observed, imply for an operator who supervises the swarm, and outline the user study needed to test it.
1 Introduction
UAV reconnaissance can shorten missions while increasing the operator’s waypoint-planning and video-monitoring work. UAV-SwarmAgent takes a single natural-language instruction and produces a multi-UAV mission through a two-tier agent architecture.
1 Introduction
- UAV supervision is demanding because each drone needs waypoints and each video stream needs attention, while workload and situation awareness degrade as teams grow.Supervisory control research also finds that the number of robots an operator can handle depends on how robot neglect time compares with service time.Source: §1 · p. 1
- For human-swarm interaction, the missing step is turning one instruction into a multi-drone mission whose duration an operator can anticipate.Prior aerial systems use language for navigation or generate a mission for one drone [7] [9] [14] [15].Source: §1 · p. 1
- A two-tier architecture has a ground-station agent plan by writing code and delegate incident verification to one agent per UAV.The system’s stated contribution also includes a 255-mission simulation and a paired study of mission time across one to five UAVs.Source: §1 · pp. 1–2
2 Related Work
Related work covers language-based supervisory control, aerial vision-language systems, and agents that act through code. UAV-SwarmAgent combines multi-drone route allocation with onboard verification and measures how mission time changes with swarm size.
Human-swarm interaction
- Language has been proposed for commanding groups at the level of intent, including formation flight and choreography [5] [10].Supervisory-control research also studies limits on how many robots one operator can handle [3, 12] and team-size effects on workload and situation awareness [2, 6].Source: §2 · p. 2
Vision-language models for UAVs
- Aerial vision-language navigation benchmarks and mission-generation systems described here focus on a single drone [7] [9] [18] [13] [14].UAV-CodeAgents grounds targets by pixel pointing for single-UAV missions [15]; UAV-SwarmAgent adds allocation across N drones and an onboard verifier.Source: §2 · p. 2
3 System
UAV-SwarmAgent uses a ground-station agent to ground inspection targets and dispatch routes, with one onboard verifier agent per UAV. Its workflow combines target selection, route allocation, and camera-frame checking.
3.1 Agents
- The Airspace Management Agent runs a ReAct loop of Python programs in a sandbox to infer targets, ground waypoints on an archival map, show them to the operator, and dispatch the swarm.One UAV agent per drone runs the onboard verifier on camera frames collected along its route.Source: §3.1 · p. 2
3.2 Grounding inspection targets
- The vision-language model receives a letterboxed 1000 × 1000 map and returns target points, with one centre point for objects and evenly spread points for areas.At most 15 points are retained; object points with equal spacing along one image axis are discarded as fabricated.Source: §3.2 · p. 2
3.3 Allocating routes
- Waypoints are assigned farthest-first to the UAV whose closed route remains shortest, then each UAV visits its points in nearest-neighbour order.Because the mission ends when the last drone returns, allocation balances the longest route rather than total distance; route length uses 1.35 m/px and cruise speed is 6 m/s.Source: §3.3 · p. 2
3.4 Verifying incidents onboard
- Each UAV’s nadir camera has a 128 × 128 px footprint, and Valen-Preview-0923, a 2B-parameter vision-language decision model, scores a frame every 16 m along the route.In one forward pass, the model returns the probability of “yes” for a scenario-specific question without generating text. Frames exceeding a per-scenario threshold (0.55–0.74) are reported; frames from all UAVs are batched on one GPU and return to the AMA with their coordinates.Source: §3.4 · pp. 2–3
4 Evaluation Setup
The evaluation pairs archival maps used for target grounding with target maps containing synthetic incidents, then tests three models across four incident scenarios and varies swarm size in a separate paired study. It measures mission success, reported-frame precision, and simulated and wall-clock time.
Scenarios and maps
- Four incident scenarios use synthetic incidents composited into target maps, while grounding uses archival reference maps of the same areas.The scenarios are structure fire, wildfire, vehicle collision, and oil spill; the operator request is one fixed sentence per scenario and never names a location.Source: §4 · p. 3
Models and protocol
- The 255-mission protocol combines a 150-mission paired fire-scenario scaling study over 1–5 UAVs with a 135-mission cross-model study at three UAVs across scenarios.Every swarm size sees the same 30 fire maps and base station; the cross-model study uses three models and includes 30 shared Gemini fire missions.Source: §4 · p. 3
Measures
- A mission succeeds when a reported frame shows the incident; precision is the share of reported frames that show it.Flight time is simulated makespan, agent time covers planning, simulation, and inference, and total time adds the two, conservatively assuming inference does not overlap flight on a real drone.Source: §4 · p. 3
Analysis
- Table 1 reports mission counts by scenario and model for N=3 UAVs, with median reported-frame precision as a separate measure.The analysis uses per-map speedup ratios, 95% bootstrap confidence intervals, and tests swarm-size effects with maps as blocks.Source: Table 1 · p. 3, §4 · p. 3
5 Results
Across the evaluated missions, the swarm located incidents in 244 of 255 missions. Flight time fell as UAV count increased, but gains flattened, while false alarms and missed incidents remained relevant to reported outcomes.
5.1 Mission success
- 244 of 255 missions (95.7%) located the incident across the evaluated scenarios and models.The cross-model study had 127 successful missions out of 135; the scaling study added 145 successes in 150 missions.Source: §5.1 · p. 3
5.2 Scaling with swarm size
- Figure 4 illustrates object targets with one waypoint and area targets with evenly spread waypoints across example missions.The collision and oil-spill examples use refined verifier questions.Source: Figure 4 · p. 4
- 505 s with one UAV fell to 363 s with five (28.1%), and Figure 5 presents mean flight, agent, and total mission times with fitted laws.The fitted Amdahl-type law assigns 34% of single-UAV flight time to the divisible share; the fixed share reflects each drone’s trip to the area and back.Source: §5.2 · p. 3, Figure 5 · p. 4
- The second UAV delivers 50% of attainable flight-time savings, while five UAVs deliver 80%.The Amdahl-type law bounds flight-time speedup at 1.52 (CI [1.40, 1.67]); a power law fits equally well, but the authors prefer the interpretable parameters.Source: §5.2 · pp. 3–4
- Total mission time fell from 657 s to 533 s (18.9%), but consecutive swarm sizes beyond two UAVs did not differ significantly.Agent time grew by 6.7 s per UAV from a 140 s intercept.Source: §5.2 · p. 4
5.3 False alarms and missed incidents
- Reported-frame precision ranged from 0.57 for wildfire to 0.95 for oil spill, and 11 missions failed: five from missed waypoint coverage and six from verifier misses.Missions with missed waypoint coverage used at most 12 of the 15 permitted waypoints, so the authors identify requesting more waypoints as a direct remedy.Source: §5.3 · p. 4
6 Discussion
The discussion frames swarm supervision as checking routes and reports rather than plotting each drone, with mission-time forecasts helping operators choose swarm size. False alarms and simulation-only evaluation constrain what can be concluded about operator use.
From plotting to checking
- A one-sentence request can produce routes for up to 5 UAVs, leaving the operator to check the waypoint overlay before dispatch and returned frames afterwards.Whether operators catch errors and at what workload remains a user-study question.Source: §6 · p. 4
Swarm size as a predictable choice
- The second UAV delivers half of the attainable saving, while a fifth buys little because travel to the area does not parallelise.The two-parameter law could forecast mission times before launch, such as about 426 s for two UAVs and 393 s for three.Source: §6 · p. 4
False alarms are an interaction problem
- A system that finds incidents in most missions can still leave operators dozens of frames to inspect, making report ranking, frame clustering, and threshold adjustment relevant interface choices.Source: §6 · p. 4
Limitations
- The evaluation uses simulated synthetic incidents and a kinematic flight model without wind, batteries, or obstacles, and includes no human participants.Verifier questions and thresholds were chosen on benchmark maps without a held-out set; each configuration ran once, and operator statements remain hypotheses.Source: §6 · p. 4
7 Conclusion and Next Steps
UAV-SwarmAgent turns one natural-language instruction into a multi-UAV reconnaissance mission, locating incidents in 95.7% of 255 simulated missions. Next steps include comparing instruction-based commanding with manual waypoint planning in a user study and moving from simulation to physical multirotors.
7 Conclusion and Next Steps
- 95.7% of 255 simulated missions located the incident, with 3 different foundation models.Source: §7 · p. 4
- The planned user study will compare instruction-based commanding with manual waypoint planning and measure planning time, workload, and error detection.Source: §7 · p. 5
- The authors plan to move from simulation to physical multirotors.Source: §7 · p. 5