OceanGym: A Benchmark Environment for Underwater Embodied Agents

1 Introduction

• OceanGym addresses an underexplored but scientifically and practically important domain, including ocean exploration, offshore development, environmental monitoring, and subsea rescue.The introduction frames robust operation in real marine conditions as a bridge between simulated research and practical deployment.Source: §1 · p. 1
• Underwater agents must contend with poor visibility, sparse and noisy heterogeneous sensing, unstable localization, dynamic currents, and extreme partial observability.Limited prior knowledge also constrains environmental modeling.Source: §1 · p. 1
• OceanGym provides eight underwater task domains and a unified framework integrating perception, memory, and action for AUV control across optical and sonar inputs.Its approximately 800m × 800m marine environment supports tasks including search, inspection, and docking, with systematic evaluation and an initial pathway toward real-world AUV skill transfer.Source: §1 · p. 2, Figure 1 · p. 2
• Experiments reveal gaps between MLLMs and human experts, especially under low visibility, with difficulties in sonar interpretation, complex-scene object distinction, and consistent long-horizon decisions.Occlusion and changing environmental conditions also expose limits in memory retention and adaptability.Source: §1 · p. 2

2 OceanGym

• OceanGym evaluates embodied agents across perception and navigation decision tasks using multimodal perception and parameterized actions.Its task taxonomy separates multi-view and context-based perception from decision tasks.Source: §2 · p. 2, Figure 2 · p. 3

2.1 OceanGym Environment

• Built on Unreal Engine 5.3, the environment models natural and engineered underwater regions, with shallow and deep settings and customizable depths, objects, and tasks.The representative depths are 50 m and 500 m.Source: §2.1 · pp. 2–3

2.2 Underwater Embodied Agents

• The MLLM-driven agent uses multimodal observations, task instructions, and contextual memory to produce perception answers or decision actions.Its explicit memory stores a sliding window of recent descriptions and actions, refreshed using a perception summary.Source: §2.2 · pp. 3–4

2.3 OceanGym Perception Tasks

• Perception tasks test object identification and localization using synchronized multi-view images or chronological observations from a fixed orientation.The dataset contains 85 scenes, and context-based tasks emphasize temporal consistency across changing observations.Source: §2.3 · pp. 4–5

2.4 OceanGym Decision Tasks

• Eight decision scenarios cover detection and tracking, including object search, pipeline inspection, and platform approaches in continuous 3D environments.Four representative tasks are also tested in low-light deep-sea conditions, with starting positions varied to test difficulty and adaptation.Source: §2.4 · p. 5

2.5 Evaluation Metrics

• Perception tasks use exact-match accuracy, while decision tasks use distance-based scores aggregated as a weighted sum of task-specific scores.For multiple-choice items, answers must match exactly as sets; decision scoring uses default distance thresholds of 30 m and 100 m.Source: §2.5 · pp. 5–6

3.1 Experimental Settings

• The experiments assess multiple MLLMs and human participants, using three runs per perception task and three starting points with three runs per point for decision tasks.Human volunteers provide perception answers and operate robots by keyboard; each trained participant performs only one task to prevent bias from prior environment exposure.Source: §3.1 · p. 6

3.2 Main Results

• In deep water, Gemini-3-Pro has the highest average perception accuracy at 37.88%, followed by GLM-4.5V at 30.15%, Qwen2.5-VL-7B at 28.48%, and MiniCPM-V-4.5 at 27.35%.In shallow, well-illuminated water, Gemini-3-Pro leads with 59.17%, followed by Qwen2.5-VL-7B at 57.12% and GLM-4.5V at 54.77%.Source: §3.2 · p. 6
• Humans outperform all models on decision tasks, scoring 100% in shallow water and 69.6% in deep water; GPT-4o-mini leads the models at 18.4% and 14.8%, respectively.Several tasks receive zero scores, indicating extreme difficulty due to small object size or time constraints.Source: §3.2 · p. 6, Table 2 · p. 7

3.3 Analysis

• Human experts benefit consistently from sonar, while MLLMs gain little from raw sonar and do not improve object identification with sonar; sonar-trained YOLO auxiliary perception improves performance.Sonar improves presence detection for both models and humans, but not model object identification.Source: §3.3 · p. 7, Figure 3 · p. 7
• Decision-task performance initially improves as operation extends from 0.5 to 3 hours, then plateaus.The paper attributes the plateau to limits in perception, memory, and reasoning, and a lack of intrinsic curiosity to explore new regions.Source: §3.3 · p. 7, Figure 5 · p. 8
• Memory transfer improves shallow-water decision-making under within-task and cross-task conditions, while only cross-task transfer gives stable gains in deep water.The authors suggest relevant prior experience can provide guidance under perceptually degraded conditions.Source: §3.3 · p. 8, Figure 6 · p. 8
• Perception failures mainly involve occlusion, multi-object ambiguity, and low illumination; decision failures mainly involve perception errors and memory forgetting.Decision-task memory failures include forgetting visited locations or prior decisions.Source: Figure 4 · p. 7, §3.3 · p. 8, Figure 7 · p. 9
• In sim-to-real tests, a simulation-trained YOLO model improves GPT-4o-mini’s interpretation of real sonar but generalizes poorly to objects absent from simulation.Aligning model actions with real operational commands and using environment memory that mirrors reality allows language prompts to control real AUVs toward targets.Source: §3.3 · pp. 8–9

4 Related Work

• Embodied-intelligence simulation platforms have been developed for terrestrial, aerial, and marine domains.Source: §4 · p. 9
• Existing embodied benchmarks cover indoor, urban, aerial, and real-world settings, but a unified benchmark for underwater multimodal embodied agents remains absent.Source: §4 · p. 9
• The emergence of MLLMs has advanced embodied agents by integrating visual understanding.Source: §4 · p. 9

• OCEANGYM is presented as the first benchmark for underwater embodied agents, with the goal of bridging simulation and reality.Source: §5 · p. 9
• The platform combines realistic environments, diverse sensing modalities, and a wide range of underwater tasks.Source: §5 · p. 9
• Experiments reveal limitations of current MLLMs in underwater settings.Source: §5 · p. 9

Limitations

• OceanGym’s simulations cannot fully capture real underwater conditions, its optical and sonar images differ from real-world images, and the large environment requires considerable computational resources.The passage names ocean currents, salinity, marine life, and geological changes as factors not accurately captured.Source: Limitations · p. 9
• The framework’s memory methods are rudimentary and lack more advanced designs.Source: Limitations · p. 10

A The Use of Large Language Models (LLMs)

• The appendix states that LLMs assist only with wording and sentence structure and are not otherwise relied on in the work.Source: A The Use of Large Language Models… · p. 15

B More Related Works

• The related-work discussion surveys simulation platforms across ground, aerial, and marine domains, and embodied-agent challenges and benchmarks across multiple settings.Source: B More Related Works · p. 15

Competitive Arena for Foundation Models

• OceanGym provides a testbed for evaluating foundation models and embodied-agent frameworks across underwater scenarios.Future work can use it to optimize prompts, improve base models and agent design, and assess trained or fine-tuned models.Source: C Applications and Future Directions… · p. 15

Memory Mechanism Evaluation

• The benchmark supports systematic memory-mechanism evaluation, with sliding-window memory used as a baseline and retrieval-based or hierarchical memory proposed for future comparison.Source: C Applications and Future Directions… · p. 16

Underwater Data Synthesis Platform

• OceanGym can generate synthetic simulation data for training perception and decision-making models, supplementing scarce real-world underwater datasets.Source: C Applications and Future Directions… · p. 16

Reinforcement Learning Testbed with Comprehensive Feedback Signals

• OceanGym offers reinforcement-learning research with diverse reward mechanisms, customizable configurations, and real-time metrics for benchmarking under varying underwater conditions.Source: C Applications and Future Directions… · p. 16

Sim-to-Real Bridge

• YOLO models trained in simulation show measurable improvements on real sonar data, but generalization is limited for objects absent from simulation.The same simulation prompts and environment memory can guide real AUVs through standardized operating commands to locate target points.Source: C Applications and Future Directions… · p. 16

D Perception Task Statistics

• 85 perception datasets comprise 55 multi-view sets and 30 context-based sets, with the context-based sets divided evenly among three subtasks.The multi-view sets include pipeline, plane, ship, tower, container-box, and no-dominant-object scenes; the context-based tasks cover following pipelines, inspecting them for damage, and scanning.Source: D Perception Task Statistics · p. 16
• YOLO-assisted GPT-4o-mini improves performance on certain simulated object classes in real sonar tests, but generalization is limited for objects not modeled in OCEANGYM.Source: Figure 9 · p. 17
• Figure 10 describes using simulated prompts and environment memory to help a real AUV locate three preset underwater objects.The figure caption says model output actions are matched to standardized MOOS-IvP or ROS commands.Source: Figure 10 · p. 17

E Decision Task Details

• Task environments use intricate 3D assets based on real-world references to represent structural and material characteristics.Figure 11 shows the perceptual input at one state.Source: E Decision Task Details · p. 17, Figure 11 · p. 18
• The robot first checks memory for target coordinates, then uses six camera feeds and reference-image comparison before systematic exploration if the target is unknown.Missions also require a 10-meter standoff from obstacles, adherence to operational boundaries, and documentation of artificial structures and special objects, excluding marine life.Source: E Decision Task Details · p. 17
• Tasks span locating a mining robot, oil drums, and electrical boxes, plus inspecting an oil pipeline in low-visibility or sediment-covered settings.Targets are identified by features such as shape, color, markings, and pipeline structure.Source: E Decision Task Details · pp. 17–18
• Other missions search for partially buried or obstructed shipwrecks and aircraft, inspect wind turbines, or approach a marked docking platform.The docking platform’s “H” symbol and regular structure serve as navigation references.Source: E Decision Task Details · pp. 17–18

F Error Type Metrics and Analysis

• Perception errors are categorized as Missing All, Spurious All, No Overlap, Partially Correct (Subset), and Partially Correct (Overlap).These auxiliary metrics characterize errors against ground-truth object presence and predictions.Source: F Error Type Metrics and Analysis · p. 18
• Sonar helps models and humans determine object presence but does not enable models to identify object identity; decision-task memory-versus-recognition error proportions vary with object size and memory-length demands.The latter comparison applies to scenarios where the target is found and normal operation proceeds.Source: F Error Type Metrics and Analysis · p. 18
• Decision-task failures are classified as Target Not Found, Found but Blocked, Recognition Error, and Memory Error.The categories distinguish ineffective search, obstructed execution, incorrect identification, and failures to maintain or recall task-relevant information.Source: F Error Type Metrics and Analysis · p. 18

G Utilization of Unreal Engine Plugins

• Using Unreal Engine plugins, the simulation models water interaction: an upward flow is configured in the environment and exerts influence on an AUV as it passes through the affected region.Source: Figure 20 · p. 19

H Prompt for OCEANGYM

• Perception prompt templates cover RGB images without sonar and sonar-supported perception using either a reference image or examples.The RGB prompts ask which listed options appear in the image.Source: Figure 21 · p. 20, H Prompt for OCEANGYM · pp. 20–21, Figure 22 · p. 21
• The navigation prompt frames the agent as an AUV control expert receiving data from six cameras and location sensors to navigate and complete tasks.Figure 24 presents the navigation prompt template.Source: Figure 24 · p. 22, H Prompt for OCEANGYM · p. 22