PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory in the Sky: Low-Altitude Question Answering with Multi-Agent Memory Aggregation

Chengyang Li, Yujie Wan, Shuai Wang, Kejiang Ye, Weijie Yuan, Boyu Zhou, Yik-Chung Wu, Chengzhong Xu, Huseyin Arslan

arXiv:2609.35431v1cs.ROcs.ITmath.IT

TL;DR

Low-altitude question answering needs a way to value distributed UAV memories for unpredictable, open-set questions while accounting for communication constraints. The paper uses GAE to measure memory utility and integrates it into MemCen resource allocation. Reported evaluations include 29 correct answers among 30 field questions, alongside results across simulated and real settings.

  • Problem

    Open-set LAQA lacks a suitable memory-oriented objective because questions are unknown at collection time and semantic similarity is not an end-to-end QA metric.

    Source: A. Related Work · p. 1, §III · p. 4

  • Method

    GAE estimates candidate-memory utility through the deployed QA pipeline, and MemCen combines that utility with wireless constraints for joint memory selection and power allocation.

    Source: §III · p. 4, §VII · p. 16

  • Results

    29 of 30 field questions were answered correctly by the complete top-5 QA pipeline, despite a supporting-pose hit rate of 86.7%.

    Source: A. Physical Multi-UAV Field Evaluation · p. 14

  • Takeaways & Limitations

    Town04, dynamic Town05, and real PMAS results support MemCen’s utility, robustness, and real-data feasibility.

    Source: §VII · p. 16

    Future work will extend the method to larger aerial networks and broader platform and task distributions.

    Source: §VII · p. 16

Abstract

from arXiv · show

This paper studies low-altitude question answering (LAQA), in which distributed unmanned aerial vehicle (UAV) memories are aggregated at a ground server to answer questions about observations over a long horizon. Unlike conventional resource allocation based on sensing, communication, control, or computation metrics, LAQA requires an explicit measure of memory value. We propose a generative adversarial exam (GAE) that uses forward simulation to evaluate memory retrieval and exam scores to quantify memory quality. This enables the downstream QA value of candidate memories to be measured and optimized without accessing the internal mechanisms of the black-box captioning, retrieval, and reasoning pipeline. Building on this metric, we develop a memory-centric (MemCen) framework that jointly selects UAVs and allocates transmit power to maximize memory quality under communication constraints. In the noise-limited regime, we derive a QoM-aware capped water-filling law that explicitly connects task utility with physical-layer power allocation. We further develop penalty successive optimization (PSO) and learning to memorize (L2M) solvers. MemCen achieves QA accuracies of 92.4% and 84.0% in CARLA Town04 and Town05 under static and dynamic communication conditions, respectively. In real-world experiments, MemCen achieves 88.5% QA accuracy on the panoramic multi-agent system (PMAS) benchmark. Finally, UAV-to-robot-dog demonstrations further validate the practical utility of the acquired memories for environmental understanding and navigation.

I. INTRODUCTION

LAQA addresses long-horizon question answering over distributed UAV observations, where conventional resource objectives and existing communication approaches do not resolve how to value memories for open-set questions. The paper proposes GAE-based memory evaluation and MemCen resource allocation, then evaluates memory reuse and system performance.

I. INTRODUCTION

  • LAQA uses distributed UAV observations to answer questions about objects and events over long horizons, beyond the short horizons and spatially imprecise outputs of conventional QA.The system aggregates UAV memories at a ground server to support agentic inference.Source: §I · p. 1, A. Related Work · p. 1

A. Related Work

  • Existing LAWN objectives emphasize coverage, throughput, safety, or computation, while semantic similarity and closed-set task optimization do not resolve memory valuation for open-set questions.The memory required for broad spatial and temporal coverage also makes standalone ground-robot approaches increasingly inefficient.Source: A. Related Work · p. 1

B. Contributions of This Work

  • GAE forward-simulates candidate-grounded questions against global memory and converts the resulting knowledge gap into a computable QoM score without requiring a differentiable QA pipeline.MemCen jointly incorporates memory utility and wireless conditions; its QoM-aware capped water-filling law and PSO/L2M solvers support allocation and inference.Source: B. Contributions of This Work · p. 2
  • The paper implements MemCen in OpenMAMS and evaluates it in Town04 and Town05, including heterogeneous UAVs, non-uniform payloads, mobility, and building blockage.It also demonstrates aerial-memory reuse by a ground robot for environmental understanding and navigation.Source: B. Contributions of This Work · p. 2

C. Outline and Notations; II. SYSTEM MODEL AND PROBLEM FORMULATION

  • The paper proceeds from the LAQA system model and GAE to cross-layer optimization, simulations, real-world experiments, and conclusions; its notation distinguishes scalars, vectors, matrices, and sets.The system aggregates UAV data at a ground server into long-horizon global memory for spatiotemporal questions.Source: C. Outline and Notations · p. 2, §II · p. 2

A. System Overview

The system collects time-stamped UAV pose-image frames, selects a subset for upload, and turns uploaded images into queryable memory. It optimizes memory quality under communication constraints, including power budgets and multiuser interference.

1) Memory Collection:

  • Each UAV collects time-stamped pose-image frames, and the system represents heterogeneous candidate payloads before selecting which UAV memories to upload.A binary selection variable indicates whether each UAV is selected for upload.Source: 1) Memory Collection · p. 2

2) Memory-Empowered Query:

  • The ground server captions uploaded images, embeds the captions, and stores time-pose-embedding tuples in a vector database for retrieval and LLM question answering.Answers are generated conditioned on the resulting memory M(X).Source: 2) Memory-Empowered Query · p. 3

B. Memory Quality Optimization

  • Selected UAVs must upload their payloads before the deadline under individual and sum-power budgets, with all active UAVs modeled as potential interferers.The system considers MRC, which does not explicitly suppress multiuser interference, and IRC, which does; IRC suits interference-limited operation when channel and covariance information are available.Source: B. Memory Quality Optimization · pp. 3–4
  • MemCen maximizes memory-conditioned QA accuracy under communication constraints, while normalized accuracy–rate optimization exposes a tunable tradeoff between memorization and communication performance.The memory-quality function lacks an explicit analytical form, motivating GAE as a computable surrogate.Source: B. Memory Quality Optimization · pp. 3–4

III. PROPOSED MEMORY QUALITY MODEL

Memory quality is defined by a candidate memory’s marginal contribution to downstream QA, rather than by its information content alone.

III. PROPOSED MEMORY QUALITY MODEL

  • A candidate memory’s utility is its marginal QA-performance improvement when added to current global memory M0, but it cannot be evaluated directly at collection time because future questions are unknown.The definition is conditioned on the deployed black-box QA pipeline and current global memory.Source: §III · p. 4

A. Generative Adversarial Exam

GAE uses candidate UAV memories to generate grounded exams and evaluates them with the deployed retrieval-and-answering pipeline, estimating each candidate’s potential utility for filling gaps in the global memory. Its overhead involves both pilot-data transmission and server-side inference.

A. Generative Adversarial Exam

  • Candidate memories generate grounded questions, which the current global memory must answer through the deployed retrieval-and-answering pipeline to obtain a normalized QoM score.A higher QoM indicates a larger knowledge gap in the global memory and higher potential utility for the candidate memory.Source: A. Generative Adversarial Exam · p. 4
  • The GAE architecture takes distributed candidate datasets and the pre-collection global memory as inputs, then evaluates candidates through a three-step pipeline.Pilot data are uploaded, captioned into grounded question–answer pairs, and tested by the retrieval-and-answering pipeline.Source: A. Generative Adversarial Exam · pp. 4–5

B. Overhead Analysis

  • Larger pilots can improve QoM-estimation and ranking reliability, but increase pilot-upload latency; small pilots upload substantially less data than complete candidate memories.Random sampling is a low-complexity pilot option, while key-frame sampling reduces the risk of missing sparse informative events.Source: A. Generative Adversarial Exam · p. 4, B. Overhead Analysis · p. 5
  • Iterative memory retrieval limits repeated processing of the full global-memory context, while concurrency reduces wall-clock latency but not total computation.Exam generation uses verified scene facts and structured question–answer pairs; unsupported guessing is prohibited and malformed outputs are invalid runs.Source: B. Overhead Analysis · p. 5

IV. MEMCEN RESOURCE ALLOCATION

The MemCen formulation accounts for pilot uploads when constraining the remaining candidate-memory transmission, and addresses the resulting joint selection-and-power problem with optimization and learned inference solvers.

IV. MEMCEN RESOURCE ALLOCATION

  • Substituting the task-driven marginal utility makes the MemCen optimization problem computable.The supplied passage introduces this step as substitution of the task-driven marginal utility derived earlier.Source: §IV · p. 5
  • The remaining-memory rate requirement uses only the residual payload, Sk − eSk, and the remaining transmission time, T − TPilot.This constraint accounts for pilot data already uploaded during GAE.Source: §IV · p. 5
  • The problem couples binary UAV selection with continuous power allocation and remains nonconvex under multiuser interference; PSO and L2M are used for stationary optimization and real-time inference, respectively.PSO solves convex surrogate subproblems, while L2M provides learned inference.Source: §IV · p. 5

A. Penalty Augmentation

PSO relaxes binary UAV selection and uses penalty-based successive optimization to obtain stationary solutions, while L2M is trained on PSO decisions to reduce online optimization overhead.

A. Penalty Augmentation

  • The relaxed selection variables are encouraged toward binary values by adding a penalty term to the objective.For a sufficiently small positive penalty parameter, the penalized and original problems are globally equivalent.Source: A. Penalty Augmentation · p. 5

B. Successive Optimization and Learning to Memorize

  • PSO replaces nonconvex terms with upper-bounding surrogate functions and iteratively solves convex subproblems around the current feasible solution.Each solution becomes the next expansion point; after convergence, thresholding produces binary UAV-selection decisions.Source: B. Successive Optimization and… · p. 6
  • Every limit point of the PSO sequence is a stationary point when the initial point is feasible, but repeated optimization may limit online responsiveness as UAV count grows.Each subproblem has 2K decision variables, and interior-point solver cost grows polynomially with K.Source: B. Successive Optimization and… · p. 6
  • L2M learns joint UAV-selection and power-allocation decisions from PSO solutions, then uses a constraint-aware decoder to mask infeasible selections and determine transmit power.A fixed Kmax-slot representation with zero-padding supports varying UAV counts up to Kmax and reduces online optimization overhead.Source: B. Successive Optimization and… · p. 6

C. QoM-Aware Capped Water Filling

Under a noise-limited condition, the relaxed MemCen problem yields a QoM-aware capped water-filling allocation that accounts for task utility and upload completion.

C. QoM-Aware Capped Water Filling

  • In the noise-limited regime, the power-allocation problem uses remaining upload time and payload after pilot transmission.The selection variables are removed after substituting the relaxed selection expression.Source: C. QoM-Aware Capped Water Filling · p. 6
  • Power is prioritized by task-weighted QoM and favored by smaller remaining payload, subject to an upload-completion cap.A UAV with zero QoM receives no power, and additional power after upload completion does not increase memory utility.Source: C. QoM-Aware Capped Water Filling · p. 7
  • If positive-utility UAVs’ aggregate caps meet or exceed the power budget, the water level exhausts that budget; otherwise, each receives capped power and leftover power adds no memory utility.Within the unsaturated region, allocation is nondecreasing in ηkQoMk.Source: C. QoM-Aware Capped Water Filling · p. 7

V. SIMULATION RESULTS

Simulation experiments use OpenMAMS with a specified captioning, embedding, retrieval, and QA stack, and compare MemCen solvers with resource-allocation and memory-selection baselines.

V. SIMULATION RESULTS

  • OpenMAMS supports multi-modal data generation, image-memory construction, question answering, and optimization algorithms for LAQA system design and evaluation.The platform is implemented in Python using CARLA and deployed on a Linux workstation with an NVIDIA RTX 5090 GPU.Source: §V · p. 7
  • The default pipeline uses Qwen3-VL-8B for image captioning, Mxbai-Embed-Large-v1 for embeddings, Milvus top-5 retrieval, and Qwen3-8B for QA.The reported objective is the sum of GAE scores weighted by UAV selection decisions.Source: §V · p. 7
  • ComCen maximizes sum rate, SenCen sensing coverage, and FairCen minimum rate, while Greedy selects UAVs by GAE score without joint channel optimization.Remember retrieves only from M0, and SemCom ranks memories by embedding similarity.Source: §V · p. 7
  • MemCen is evaluated with PSO, L2M, relax-and-round selection, and DQN with a feasibility mask.PSO solves MemCen by penalty successive optimization; L2M predicts decisions using a learned surrogate.Source: §V · p. 7

A. Town04 Evaluation

Town04 experiments specify the simulation and question setup, then test whether GAE distinguishes heterogeneous memories and selects useful additional memory for downstream QA.

A. Town04 Evaluation

  • Town04 tests randomly place 10 objects and generate 30 questions per realization across object presence, location, and observing-UAV identification.Each UAV records 1,050 frames at 35 FPS; the GAE exam uses 5 or 10 questions, with 10 images per UAV for pilot sampling.Source: A. Town04 Evaluation · p. 7
  • The air-to-ground communication setup uses 10 MHz bandwidth, a 600 s wireless collection budget, 300 mW total transmit-power budget, and a 100 mW maximum per-UAV transmit power.For dense-building scenarios, the air-to-ground channel follows Rayleigh fading, with 256 ground-server antennas and MRC unless otherwise stated. Quantitative results are averaged over 20 independent simulation runs.Source: A. Town04 Evaluation · pp. 7–8

B. Validation of the GAE

  • GAE scores increase from UAV 1 to UAV 4 with observation richness, and both QA models distinguish memory quality, with Qwen3-14B achieving higher accuracy than Qwen3-8B.Table I reports average GAE scores and QA accuracies for the two models over 20 random runs.Source: B. Validation of the GAE · pp. 8–9, Table I · p. 8
  • GAE selects UAV 4 and achieves 70% downstream QA accuracy, while SemCom selects UAV 2 and achieves 48%.The comparison uses each memory union with historical memory M0; the reported answers for four listed objects were all correct.Source: Figure 5 · p. 8, B. Validation of the GAE · p. 9

C. Evaluation of PSO and L2M

MemCen outperforms the supplied Town04 baselines on QA and memory quality, while solver tests show that PSO converges quickly and L2M sharply reduces execution time. The evaluation also examines resource-allocation trade-offs and performance under interference-limited channels.

C. Evaluation of PSO and L2M

  • For β ≤ 0.5, tested final mean zero-one residuals range from 0.073 to 0.080; β = 0.5 yields 0.080, while β ≥ 1 exceeds 0.1.The 0.1 threshold is the near-binary criterion for K = 10.Source: C. Evaluation of PSO and L2M · p. 9
  • Within five iterations, PSO’s normalized objective gap approaches zero across N = 128, 256, and 512, without materially slower convergence as N increases.From iterations 4 to 5, relative objective changes are 0.53%, 0.009%, and 0.10%, respectively.Source: C. Evaluation of PSO and L2M · p. 9
  • L2M retains 95.0% of PSO’s average sum GAE score while reducing execution time from 4.60 s to 0.04 s, a 115× speedup.Relax-Round incurs a 15.7% performance loss relative to PSO.Source: C. Evaluation of PSO and L2M · p. 9
  • Without fine-tuning, L2M retains 100.00%, 99.80%, 98.93%, and 97.46% of branch-and-bound optimal QoM at K = 5, 10, 15, and 20.Selected-UAV counts closely match the optimum, and all decoded decisions satisfy communications constraints.Source: C. Evaluation of PSO and L2M · p. 9

D. Comparison with Existing Benchmarks

E. Dynamic and Heterogeneous Town05 Evaluation

In dynamic, heterogeneous Town05 conditions, MemCen achieves the highest reported QA accuracy while adapting to mobility, blockage, and unequal payloads. Physical PMAS experiments further evaluate memory selection, retrieval, and QA, including sensitivity to sampling and model choices.

E. Dynamic and Heterogeneous Town05 Evaluation

  • Switching from urban inspection to suburban rescue changes the urban-route aggregate weight from 0.354 to 0.060, reweighting the same scene and memory interface without platform-specific retraining.The setting combines moving UAVs, building blockage, heterogeneous platforms, and task-dependent route weights.Source: E. Dynamic and Heterogeneous Town05… · pp. 11–12, Figure 11 · p. 12
  • 84.0% is MemCen’s highest Town05 QA accuracy; it completes all selected uploads and reduces QA-accuracy standard deviation from 16.5% to 10.5%.Its selected set averages 1.70 fixed-wing and 5.00 multirotor UAVs under heterogeneous payloads and dynamic channels.Source: E. Dynamic and Heterogeneous Town05… · p. 12
  • As fixed-wing workloads rise from 1,000 to 2,000 frames, admitted UAVs fall from 9.00 to 6.70, aggregate QoM from 3.390 to 2.775, and QA accuracy from 93.0% to 84.0%.All selected uploads remain feasible; the higher sum rate under larger workloads results from fewer active UAVs reducing multiuser interference.Source: E. Dynamic and Heterogeneous Town05… · p. 12

A. Physical Multi-UAV Field Evaluation

  • On real PMAS data, MemCen improves mean QA accuracy by at least 12.0 percentage points over every baseline and has 54.7% higher sum QoM than SenCen at the same average 1.90 uploads.The experiment uses three panoramic UAVs and evaluates aerial memories in a shared 3D coordinate frame.Source: A. Physical Multi-UAV Field Evaluation · pp. 12–13, Table VII · p. 13, Figure 12 · p. 13
  • With UAV 1 as initial memory, MemCen selects UAV 2, whose combination reaches 75% QA accuracy versus 70% for UAVs 1 and 3 and 50% for UAV 1 alone.UAV 2 has lower trajectory and semantic overlap with UAV 1 than UAV 3, and higher marginal QoM.Source: A. Physical Multi-UAV Field Evaluation · pp. 13–14, Figure 14 · p. 14
  • Key-frame sampling meets both pilot reliability criteria at 2% sampling, with 0.939 Spearman correlation and 88.50% QA accuracy; random sampling meets them at 10%.These minimum observed ratios are specific to the tested scene and frame rate.Source: A. Physical Multi-UAV Field Evaluation · p. 14, Figure 15 · p. 14
  • Across tested model choices, absolute QoM values vary, but memory selections remain stable in most settings and are mainly sensitive when candidate QoM scores are similar.With captions fixed, changing the embedding model preserves all three selections for the 4B and 8B answering models; the 30B model has one tie and one changed selection.Source: A. Physical Multi-UAV Field Evaluation · p. 14, Table VIII · p. 14
  • Top-5 retrieval finds a supporting pose for 26 of 30 questions, yet the complete QA pipeline answers 29 correctly, showing that an annotated pose is not always necessary.Retrieval failures are still treated as part of end-to-end behavior in pipeline-conditioned QoM evaluation.Source: A. Physical Multi-UAV Field Evaluation · p. 14

B. Drone-to-Dog Collaboration

The drone-to-dog demonstration uses aggregated UAV memory to answer a venue query and guide the robot to the basketball court. The conclusion frames the broader results as support for MemCen’s utility, robustness, and real-data feasibility, while identifying larger networks and broader platform and task distributions as future work.

B. Drone-to-Dog Collaboration

  • In campus views, the drone recognizes a park, a statue, and a residential building.The observations come from the segmented front-camera video described in the experiment.Source: B. Drone-to-Dog Collaboration · p. 15
  • The system selects segment 4 for aggregation because it has the highest GAE score, while segment 1’s similar views to initial memory M0 yield high GAE QA accuracy.The authors present this selection as evidence that GAE identifies fresh memory in real-world data.Source: B. Drone-to-Dog Collaboration · pp. 15–16
  • The robot uses a position retrieved from UAV memory, transforms it into dog coordinates, and follows a planned path to reach the basketball court.The demonstration connects memory-based agentic inference to the path planner and locomotion controller.Source: Figure 19 · p. 15, B. Drone-to-Dog Collaboration · p. 16

VII. CONCLUSION

  • The paper concludes that Town04, dynamic Town05, and real PMAS results support MemCen’s utility, robustness, and real-data feasibility.Future work will extend the method to larger aerial networks and broader platform and task distributions.Source: §VII · p. 16
Loading 2609.35431v1…