Source-linked AI summary
A Cognitive Architecture for Shared Autonomy in AUV Operations
Niamh Ellis, Thi Tran, Ignacio Carlucho, Yvan R. Petillot
TL;DR
ROV operators remain necessary but can experience low situational awareness and high workload, motivating decision support for shared autonomy. The paper combines an ontology with multiple role-specific LLMs to evaluate mission feasibility, plan missions, and execute them in simulation. GPT-OSS performs best for feasibility assessment, planning, and execution, while Qwen2.5 best identifies mission types from natural language.
Problem
ROV operators often experience low situational awareness and high workload, while full embedded autonomy remains distant and ontology-only systems struggle with implicit knowledge and human-level reasoning.
Method
The framework combines an ontology with multiple role-specific LLMs grounded in domain knowledge to support mission evaluation, planning, execution, and operator oversight.
Results
GPT-OSS performs best for mission evaluation, planning, and execution, while Qwen2.5 performs best at identifying mission types from natural-language input.
Takeaways & Limitations
The framework supports shared-autonomy operations by grounding decisions in relevant domain knowledge while using LLM reasoning across mission stages.
Abstract
from arXiv · showhide
Operators remain essential to Remotely Operated Vehicle (ROV) operation, yet often suffer from low situational awareness and high workload, both of which negatively affect safety. This paper presents a cognitive architecture consisting of an ontology and multiple Large Language Models (LLMs) to assist the operator at all stages of the mission. Each LLM is grounded with domain-specific information from the ontology and given a simple role to create a system that can support the operator at all stages of an operation. We are aiming to prove that using the two together will allow decisions to be grounded in the relevant domain knowledge, but also benefit from the reasoning capabilities of the LLM. Our framework determines if a mission is possible for a given Unmanned Underwater Vehicle (UUV), performs mission planning, and executes a given mission in simulation. The operator can be involved in planning and execution, ensuring the resulting plan is valid and that the vehicle behaves safely during execution. We compare different LLMs, Llama3, GPT-OSS, and Qwen2.5, to determine which are best suited to the different roles within our framework. We find that GPT-OSS performs best for feasibility assessment, planning, and execution, while Qwen2.5 is best suited to identifying mission types from natural language input.
I. INTRODUCTION
The paper proposes a shared-autonomy cognitive architecture that combines ontology-grounded, role-specific LLMs to support operators from mission evaluation through execution. It addresses operator workload and situational-awareness problems while evaluating three LLMs for different framework roles.
- Operators remain essential in ROV missions but often face low situational awareness and high workload, affecting safety and increasing mission-critical error risk.
- Existing ontology-based approaches can struggle with implicit domain knowledge and human-level logical reasoning in complex, dynamic marine scenarios.
- The proposed architecture combines an ontology with multiple role-specific LLMs, grounding each model in domain information while decomposing complex tasks into simpler tasks.
- The framework aims to support operators across mission planning, execution, and unexpected situations using domain knowledge and LLM reasoning together.
- The study evaluates Llama3, GPT-OSS, and Qwen2.5 to identify models best suited to mission evaluation, planning, execution, and related roles.
II. RELATED WORK
Prior marine autonomous-system approaches demonstrate the value of ontology-based domain knowledge but provide limited logical reasoning for dynamic missions. Recent work therefore uses LLMs to address this reasoning gap.
- Marine autonomous systems have used ontologies to store and share domain knowledge between vehicles, with some extensions adding SWRL-based reasoning rules.
- Knowledge-representation approaches remain limited in logical reasoning, which is treated as important for dynamic missions and motivates recent LLM-based approaches.
B. Large Language Models
Related robotics research combines LLMs with diagnostics systems, vision-language models, ROS, knowledge graphs, and safety mechanisms. The paper distinguishes its framework by assessing vehicle capability before planning and replanning missions.
- A two-LLM diagnostic system combines a vector database and digital twin to help operators determine causes of unexpected UUV behaviour.
- Other systems ground LLM planners with visual information, XML domain knowledge, or ROS-based natural-language control.
- Knowledge-graph approaches address safe command execution or update task sequences with domain knowledge, but may omit vehicle capabilities and operational effects.
III. METHODOLOGY
The system architecture comprises an ontology, collaborative agentic LLMs, and an LLM supervisor, with ROS coordinating communication and adding grounding. The agents evaluate, plan, and execute missions.
- The architecture contains an ontology, agentic LLMs, and an LLM supervisor that collaboratively support mission evaluation, planning, and execution.
- ROS channels communication between the separate LLM agents and provides additional grounding for their collaboration.
A. Ontology
The ontology stores domain knowledge for mission evaluation, planning, and execution, grounding LLM decisions in vehicle capabilities, mission requirements, actions, constraints, and software interfaces.
- A. Ontology: The ontology organizes knowledge for mission evaluation, planning, and execution.It represents vehicle capabilities, mission types, actions, constraints, and ROS action interfaces.
- A. Ontology: Vehicle instances link hardware, software, and capabilities to support feasibility assessment against mission requirements.Mission types are related to required capabilities, allowing the system to compare a specific vehicle configuration with a requested mission.
- A. Ontology: SWRL rules infer capabilities such as path planning from localisation and an onboard path-planning algorithm.The mission evaluation LLM retrieves inferred vehicle capabilities and mission requirements rather than reasoning directly about hardware or software.
- A. Ontology: Planning knowledge links mission actions to preconditions, postconditions, software, vehicle constraints, and environmental information.These relationships support plans that are valid for the vehicle and its operating environment.
- A. Ontology: Execution knowledge stores ROS actions and request parameters, distinguishing mission-dependent values supplied by the LLM from static ontology values.For example, the LLM may adjust inspection standoff distance, while the ontology supplies a structure’s location.
- A. Ontology: Fig. 2 depicts mission types, actions, and software relationships using different line styles and colours.Blue lines indicate subclass relationships, purple lines indicate class membership, and dashed lines represent relationships between instances.
B. LLM Agents
The framework divides complex operator queries among locally hosted LLM agents with distinct roles for mission evaluation, planning, execution, and diagnostics.
- B. LLM Agents: Individual LLM agents perform simple role-specific tasks that together handle complex user queries.Each LLM is hosted locally via Ollama.
- B. LLM Agents: The mission-evaluation role identifies the mission type most relevant to the user’s natural-language description.It receives the possible mission types from the ontology and the user’s description.
- B. LLM Agents: The planning role generates an action plan collaboratively with the user using mission actions and their preconditions and postconditions.The user can request improvements before execution, such as adding steps or revising the plan.
- B. LLM Agents: Execution uses a tool-based LLM architecture to call environmental actions, while ROS services and actions support short- and long-duration behaviours.ROS actions allow feedback or cancellation for longer-lasting behaviours such as path following.
- B. LLM Agents: A diagnostics LLM detects and analyses unexpected events and can suggest mitigations or mission replanning with the user.Its diagnosis uses ontology information and web searches, and it can help cancel in-progress actions when needed.
IV. RESULTS
The framework was tested in Stonefish with a laboratory ROV and three LLMs across mission evaluation, planning, and execution scenarios.
- IV. RESULTS: The framework was tested in Stonefish using a laboratory ROV equipped for inspection in good environmental conditions.The vehicle has ROS communication, a stereo camera, sonar, and an underwater manipulator.
- IV. RESULTS: Llama3, Qwen2.5, and GPT-OSS were evaluated across mission evaluation, mission planning, and mission execution components.The evaluation covered different scenarios using the three LLMs.
A. Mission Evaluation Results
Mission evaluation tests varied vehicle capabilities and ontology access, while mission-type identification addresses overconfident feasibility judgments for partially capable vehicles.
- A. Mission Evaluation Results: Three vehicle configurations represented fully possible, partially possible, and impossible inspection-and-anode-testing missions.The configurations varied whether required inspection capabilities and an onboard manipulator were available.
- A. Mission Evaluation Results: Each LLM was tested five times with ontology mission requirements and five additional times without them.The same natural-language mission description was used to obtain the three possible feasibility outcomes.
- A. Mission Evaluation Results: Without ontology requirements, the LLMs often handled fully possible and impossible configurations but struggled with partially possible configurations.They could overestimate feasibility when essential hardware, such as a manipulator, was missing.
- A. Mission Evaluation Results: Without ontology grounding, models sometimes identified partial capability but named the missing component incorrectly.The authors report this as validating ontology grounding for LLMs.
- A. Mission Evaluation Results: Mission-type identification was evaluated separately because overconfident assignment of Fully capable could occur for partially capable vehicles.The same mission description was supplied for the mission-type evaluation reported in Table II.
B. Mission Planning Results
The framework evaluates whether generated mission plans are valid across prerequisite satisfaction, mission completion, and redundancy or contradiction. GPT-OSS performs best overall, but results for anode testing indicate that further planning development is needed.
- The evaluation checked whether plans satisfied necessary preconditions, completed mission objectives, and avoided redundant or contradictory steps.Tests used two mission descriptions, including an inspection generating a 3D map.
- Five tests were run for each mission description, including an anode-testing mission and a simpler inspection mission.
- GPT-OSS was the only model producing plans that would successfully complete the anode inspection.
- Overall results for the anode-testing mission suggest that additional work is needed on the planning component.
- 100% of GPT-OSS inspection plans met preconditions and successfully completed the mission.Llama3 achieved comparative results on these metrics but generated multiple redundant steps.
C. Mission Execution Results
Mission execution used a GPT-OSS-generated inspection plan implemented through ROS actions and services in Stonefish simulation. Execution evaluation considered action calls, incorrect calls, and required human corrections, while Llama3 results were omitted because of persistent execution difficulties.
- The inspection plan executed four behaviors: SLAM initialization, coverage-path generation, path following, and movement near the surface.The implementation used ROS actions and services for the required behaviors.
- Execution was evaluated by the number of actions called, incorrect actions called, and human corrections required.Human correction included wrong names, parameters, timing, or missing parameters.
- Llama3 results were omitted because it consistently struggled to execute the mission and often produced incorrectly formatted output.
- GPT-OSS performs best across mission evaluation, planning, and execution, whereas Qwen2.5 performs best at identifying mission types.The authors theorize that GPT-OSS’s reasoning behavior and Qwen2.5’s handling of mission wording contribute to this division of roles.
V. CONCLUSION AND FUTURE WORK
The framework evaluates vehicle-mission feasibility, generates simple plans, and executes them under perfect conditions. GPT-OSS is best for feasibility assessment, planning, and execution, while Qwen2.5 is best for mission-type identification; future work targets real-world testing and diagnostics.
- The framework evaluates vehicle capability, generates a simple mission plan, and executes it under perfect conditions.
- Qwen2.5 is best suited to identifying mission types, while GPT-OSS is best suited to feasibility assessment, planning, and execution.
- Future work will test the framework in the real world and further develop components, particularly diagnostics.
- Diagnostics development is intended to support more complex scenarios, failure adaptation, and testing under unforeseen circumstances.