Source-linked AI summary
Offline Language-Based Support System for Tactical Decision-Making in Air Combat Operations
Joao P. A. Dantas
TL;DR
Pilots need quick access to tactical information in lengthy manuals, but manual lookup can be impractical and cloud-dependent language tools may not suit sensitive or disconnected settings. MATE uses offline retrieval-augmented generation to answer questions from doctrinal manuals. In an exploratory pilot evaluation, answers were rated adequate for most questions, while retrieval was identified as the main improvement area.
Problem
Pilots and mission planners may need fast access to tactical information in lengthy manuals, while manual lookup can be impractical and cloud-dependent tools limit use in sensitive or disconnected settings.
Source: §1 · p. 1Method
Results
Answer ratings averaged 4.00–4.40 on a five-point scale, Mistral NeMo 12B had the highest average, and retrieval relevance and integration were the lowest-rated tool aspects at 3.67.
Source: §4 · pp. 4–5Takeaways & Limitations
The authors report that pilots considered response time fully adequate compared with manual search and identify retrieval as the main component to improve.
Source: §4 · p. 5The evaluation was exploratory: it involved three pilots and one chapter of one manual, and did not support statistical model comparisons or meaningful inter-rater agreement analysis.
Source: §4 · p. 5
Abstract
from arXiv · showhide
An offline language-based support tool is presented to assist pilot training and tactical decision-making in air combat operations. The system, named MATE (Mission Assistant for Training and Evaluation), applies retrieval-augmented generation (RAG) to answer natural language questions using content extracted from doctrinal and tactical manuals. Built entirely with open-source components, it performs text preprocessing, embedding generation, semantic search, and language generation locally, ensuring full offline functionality. The tool was evaluated by three Brazilian Air Force fighter pilots, who rated the answers produced with three open-weight language models (Llama 3.1 8B, Qwen 2.5 14B, and Mistral NeMo 12B) to five tactical questions derived from a publicly available air combat manual. Average ratings ranged from 4.00 to 4.40 on a five-point scale, with Mistral NeMo 12B obtaining the highest score, and the pilots rated the response time of the tool as fully adequate compared to manual search. The relevance of the retrieved content was identified as the main point for improvement. Potential applications are discussed in onboard mission systems, flight simulators, and training environments.
1 Introduction
Pilots and mission planners need fast access to tactical information in lengthy manuals, but manual lookup can be slow and impractical, while cloud-dependent language tools limit use in sensitive or disconnected settings. MATE addresses this gap with an offline, open-source RAG tool for doctrinal manuals, validated by fighter pilots.
1 Introduction
- Manual consultation of lengthy tactical manuals can be slow and impractical when pilots and mission planners need information quickly, including in flight.The paper notes that internet access may be unavailable or disallowed in these settings.Source: §1 · p. 1
- Cloud-dependent language tools limit natural-language document access in sensitive or disconnected environments such as military aircraft and secure simulators.Source: §1 · p. 1
- MATE uses retrieval-augmented generation to answer natural-language questions from doctrinal manuals with a local language model, open-source tools, and fully offline operation.The system is modular and described as adaptable to different document types.Source: §1 · p. 1
- The paper reports expert validation in which Brazilian Air Force fighter pilots rated answers from three open-weight language models.The tool is presented as supporting access to sensitive documents that should not be processed online.Source: §1 · p. 2
2 Related Work
Prior work has applied NLP to defense retrieval, information processing, intelligence analysis, and air-combat interfaces. These examples range from early information-retrieval initiatives to systems for human–AI collaboration and aircraft voice control.
2 Related Work
- Military NLP work includes applications to decision-making, intelligence analysis, and operational efficiency, with DARPA’s TIPSTER program targeting defense information retrieval and extraction [Defense Advanced Research Projects Agency, 1991].Source: §2 · p. 2
- Field-oriented examples include secure mobile applications from TransApps and MiTAP’s NLP-based gathering, translation, and presentation of global-event information [Defense Advanced Research Projects Agency, 2010].Source: §2 · p. 2
- NLP has been used to identify disinformation for U.S. Special Operations Command, while COA-GPT generates courses of action for military operations [Goecks and Waytowich, 2024].Source: §2 · p. 2
- Air-combat applications include DARPA’s human–AI collaboration program for pilots managing unmanned aircraft and F-35 voice interfaces for communications and navigation [Schutte, 2007].Source: §2 · p. 2
3 Methodology
MATE combines local document retrieval and language generation in a modular, offline system. Its pipeline indexes manuals, retrieves relevant segments for each question, and passes the resulting context to a locally deployed model through an interface.
3 Methodology
- MATE’s five-part architecture comprises document indexing, semantic retrieval, language-model interaction, a user interface, and offline deployment.Source: §3 · p. 2
- Manuals are converted to text and split into overlapping 1,200-character segments; 384-dimensional normalized embeddings are stored in a FAISS inner-product index for cosine-similarity search.The segments overlap by 250 characters, and metadata links each segment to its source document.Source: §3 · p. 2
- For each natural-language question, MATE retrieves the five nearest segments and concatenates them in similarity order into a context capped at 4,000 characters.Source: §3 · p. 2
- The context and question are sent to a locally deployed model through Ollama; the evaluation used Llama 3.1 8B, Qwen 2.5 14B, and Mistral NeMo 12B.A FastAPI backend constructs the prompt and streams the response to the interface.Source: §3 · pp. 2–3
- The Tkinter interface accepts natural-language questions and displays answers, while all components and models are stored locally so runtime needs no internet connection.Optional voice input was disabled during the evaluation.Source: §3 · p. 3
4 Expert Validation
Three Brazilian Air Force fighter pilots assessed MATE using five questions drawn from a publicly available air-combat manual. They rated answers from all three models as adequate for most questions, while retrieval-related ratings were the lowest and the authors characterize the findings as exploratory.
4 Expert Validation
- The five questions, derived from one publicly available manual chapter, covered mutual support and the evolution from Fighting Wing through Double Attack to Loose Deuce.The knowledge base used Chapter 5, “Section Tactics: Two-versus-One,” of Shaw’s manual because classified documents could not be disclosed.Source: §4 · pp. 3–4
- Each pilot rated 15 answers—five questions with three models—and five tool-level aspects on a 5-point Likert scale.The panel comprised three fighter pilots with an average of 6.7 years of operational experience and 933 flight hours.Source: §4 · p. 4
- Answer ratings averaged 4.00–4.40 on a 1–5 scale; Mistral NeMo 12B had the highest average, and Q3 received a rating of 3 from every pilot for every model.For Q4, Llama 3.1 8B received 3.33 and Mistral NeMo 12B received 4.67; the authors suggest the model-invariant Q3 pattern more likely reflects retrieved content than generation.Source: Table 1 · p. 4, §4 · p. 4
- Retrieval relevance and integration received the lowest tool-level ratings at 3.67, while terminological accuracy and clarity were rated 4.33; response time received the maximum rating from all pilots.The authors note that the 4,000-character context limit lets only about three of five retrieved segments reach the model in full, potentially excluding lower-ranked relevant passages.Source: Table 2 · p. 5, §4 · p. 5
- The authors characterize the evaluation as exploratory because it used a small panel and one chapter of one manual, with too few answers for statistical model comparisons or meaningful inter-rater agreement analysis.Source: §4 · p. 5
5 Conclusion and Future Work
MATE combines manual retrieval with a local language model to support air-combat decision-making and training offline. Pilot ratings found its answers and response time adequate, while identifying retrieved-content relevance and integration as areas for improvement.
5 Conclusion and Future Work
- MATE provides offline natural-language answers from doctrinal manuals using open-source tools and a modular architecture suited to simulators and mission-planning systems.The authors describe possible support during training, operational missions, and post-mission debriefings, while noting the system is still in an early stage.Source: §5 · p. 5
- Three Brazilian Air Force fighter pilots rated answers adequate for most tactical questions, with Mistral NeMo 12B highest, and response time fully adequate versus manual search.The lowest ratings were associated with relevance and integration of retrieved content.Source: §5 · p. 5
- Future evaluation will include more pilots, questions, scenarios, and manuals, while retrieval improvements target segmentation, passage selection, and context supplied to the model.Source: §5 · p. 5
- Planned work also includes hardware for larger models, dialogue history and simulation integration, and voice-module optimization because it is currently too heavy for routine flight or simulator use.Source: §5 · p. 5
Code Availability
The MATE source code and evaluation configuration are publicly available, but the copyrighted reference manual is not distributed with them.
Code Availability
- The public repository contains MATE's indexing, retrieval, backend, and interface components plus the evaluation configuration; the copyrighted reference manual is excluded.Source: Code Availability · p. 6