Source-linked AI summary
Context Aware AI Assistant and AR Interface for Lunar Extravehicular Activity (EVA) Procedural Guidance
Rodrigo Gallardo, Qilmeg Doudatcz, Ganit Goldstein, Ilkyaz Sarimehmetoglu, Sergio Mutis, Alexander Htet Kyaw, Anita Lin, Clara Emmerling, Berfin Ataman, Skylar Tibbits
TL;DR
During lunar EVAs, astronauts need procedural guidance that is accessible despite high cognitive demands and limited attention. GAIN-AI grounds an LLM in structured EVA context and presents Goal-Task-Verification guidance, scoring 10.0/10 in nominal scenarios and 8.15/10 in single-fault scenarios while degrading under greater complexity.
Problem
Existing EVA procedures are difficult to use in real time, while ungrounded LLMs can misinterpret intent in multi-fault and boundary-threshold conditions.
Method
GAIN-AI grounds an LLM with JSON-encoded procedures, telemetry, and error-handling data, then restructures outputs into compact Goal-Task-Verification AR units.
Results
Performance is strong on nominal and single-fault scenarios but degrades with complexity, reaching 10.0/10 and 8.15/10 respectively.
Takeaways & Limitations
The system detects anomalies relatively consistently but is less reliable at prescribing precise corrective actions, especially in complex scenarios.
Takeaways & Limitations
The synthetic evaluation does not measure effects on real user performance or cognitive workload, and integrated-system results are not yet available.
Abstract
from arXiv · showhide
As human space exploration returns to the Moon, astronauts need rapid access to procedural information during extravehicular activities (EVAs), where attention is divided across navigation, repair tasks, tool handling, and environmental risk. The challenge is not the absence of information, but surfacing the right information at the right moment. We present GAIN-AI (Guided Assistant for Intelligent Navigation), a context-aware AI assistant and minimal heads-up interface for procedural guidance in simulated lunar EVA. The system operates in two layers. The first grounds a large language model with structured context: EVA procedure documents, live telemetry data, and error-handling protocols encoded as JSON. The second restructures that output into three compact units for AR display: Goal, Task, and Verification. Evaluated on 111 synthetic EVA scenarios, the system scores 10.0/10 on nominal conditions and 8.15/10 on single-fault scenarios, with performance degrading on multi-fault and boundary-threshold cases.
1 Introduction
Lunar EVA demands sustained attention across objectives, hazards, and multi-step procedures, while comprehensive procedures remain difficult to use as real-time interfaces. GAIN-AI addresses this gap by grounding an LLM with structured JSON context and presenting actionable guidance through Goal-Task-Verification AR cards.
- Motivation: EVA is cognitively demanding because astronauts must manage mission objectives, environmental hazards, and multi-step procedures under physical constraint.High cognitive workload is a documented safety risk.
- Prior work: AR-assisted EVA systems support biometrics, navigation, and communication, while LLM-AR integration can reduce cognitive load during time-sensitive operations [Thomas et al. 2020; Zhuang et al. 2025; Xu et al. 2025].These capabilities motivate heads-up procedural support during EVA.
- Problem: Domain-specific grounding is needed because standard LLMs can misinterpret user intent, particularly in multi-fault or near-boundary-threshold situations [Gallardo et al. 2025; Kyaw et al. 2025].The supplied passage identifies these conditions as a specific concern for ungrounded models.
- Contribution: GAIN-AI combines a general-purpose LLM grounded with structured JSON context and a Goal-Task-Verification AR display that restructures output into discrete, actionable cards.The system reshapes official procedures to support action under constraint rather than replacing them.
2 System Overview
GAIN-AI combines a context-grounded LLM with a structured AR delivery layer for procedural guidance during simulated lunar EVA. It uses procedure, telemetry, and error-handling JSON to generate sequential Goal–Task–Verification units on a HoloLens 2 display.
- System architecture: GAIN-AI parses procedure documents into Goal–Task–Verification units and progressively renders them on an AR heads-up display.The system architecture separates AI-layer parsing from AR-layer rendering.
- Layer 1: Context-grounded LLM: The context-grounded LLM receives procedure data, live suit and environmental telemetry, and error-handling data specifying fault conditions, severity thresholds, and corrective actions.The structured context is provided at inference time across 111 synthetic EVA scenarios spanning nominal and fault-related conditions.
- Layer 2: Structured AR delivery: Grounded output is delivered as sequential Goal, Task, and Verification cards on Microsoft HoloLens 2, one card at a time with user-controlled navigation [Zhao et al. 2025].The units define the current objective, required action, and observable completion evidence while preserving operator agency.
3 Interface Design
GAIN-AI uses a minimal three-field card interface designed for EVA conditions that restrict reading, interaction, and attention. Each card prioritizes the current action while keeping completion verification immediately accessible.
- Card Structure: Each card contains exactly three labeled fields—Goal, Task, and Verification—with the current action prioritized and verification immediately accessible (Fig. 3).Goal gives the high-level objective, Task gives one imperative physical instruction, and Verification gives the observable completion condition.
- Design Motivation: The interface addresses limited visibility, gloved interaction, and divided attention by avoiding dense displays that degrade situational awareness and can increase cognitive load [Gupta and Kyaw 2025].The design motivation is grounded in pressurized helmets, restricted fields of view, gloves that preclude fine motor interaction, and physical exertion.
- Verification Design: Verification is elevated to a primary field because completion must be confirmed through an observable condition such as a sensor reading, physical state, or system indicator.The passage identifies missed verifications as a common source of procedural error.
4 Evaluation
GAIN-AI was evaluated with full structured context injection across 111 synthetic EVA scenarios using a 10-point score covering detection, action, urgency, and triage. Performance was perfect on nominal cases but declined as faults, vehicle errors, and threshold-sensitive procedures increased complexity, with anomaly detection stronger than action execution.
- 4 Evaluation: The evaluation covered 111 synthetic scenarios in five categories using error detection, correct action, urgency calibration, and triage ordering for a maximum of 10 points [Chang et al. 2023].The tested model was gemini-2.5-flash-lite with full structured context injection, selected for multimodal capabilities and latency profile.
- 4 Evaluation: 10.0/10 on nominal scenarios and 8.15/10 on single-fault scenarios declined to 7.73/10 for multi-fault, 3.96/10 for LTV errors, and 5.24/10 for boundary cases.Table 1 reports the 10-point evaluation across EVA scenario categories; multi-fault performance fell because of action-sequencing failures.
- 4 Evaluation: Error detection averaged 2.52/3 across categories, while correct action averaged 1.74/3, indicating stronger anomaly identification than downstream procedural execution.The passage reports that correct action drops sharply relative to detection across scenario categories.
- 4 Evaluation: The 13-step NAV restart sequence required exact switch states, indicator colors, and reset timing, while boundary cases depended on precise pressure thresholds that the model rounded or misordered.Examples included tank pressures above 3000 psi or below 10 psi.
5 Limitations and Future Work
The work is an early design prototype whose evaluation separates LLM logic and UI design, so integrated system performance remains pending. Future work will add a no-injection baseline, live telemetry, comparative user studies, and eventually multimodal hands-free input.
- Limitations: Integrated system performance remains unresolved because the current evaluation separates LLM logic and UI design, while complete human-factors study results are forthcoming.The prototype is one subsystem within a larger EVA system that includes a camera-free gesture-recognition glove.
- Future Work: Future work will add a no-injection baseline to complete the comparison and integrate live telemetry for real-time updates.
- Future Work: User studies will compare the GTV display against conventional checklists, with multimodal hands-free input targeted longer term.