Source-linked AI summary
OceanPile: A Large-Scale Multimodal Ocean Corpus for Foundation Models
Yida Xue, Ningyu Zhang, Tingwei Wu, Zhe Ma, Daxiong Ji, Zhao Wang, Guozhou Zheng, Huajun Chen
TL;DR
Marine AI is constrained by fragmented, heterogeneous ocean data and limited multimodal alignment. OceanPile addresses this gap with an aligned corpus, knowledge-guided instruction data, and evaluation benchmarks, significantly improving MLLM performance on marine tasks.
Problem
Ocean data remain fragmented, noisy, weakly labeled, and poorly aligned across modalities, while existing marine datasets lack comprehensive multimodal alignment and interdisciplinary coverage.
Method
OceanPile integrates authoritative multimodal data, synthesizes knowledge-augmented instructions through a hierarchical Ocean Concept Knowledge Graph, and provides expert-validated textual and multimodal benchmarks.
Results
32.59 overall multimodal score: OceanPile significantly enhances MLLM performance across marine intelligence tasks.
Takeaways & Limitations
OceanPile establishes an open-access foundation for developing capable, domain-specific MLLMs for marine AI.
Abstract
from arXiv · showhide
The vast and underexplored ocean plays a critical role in regulating global climate and supporting marine biodiversity, yet artificial intelligence has so far delivered limited impact in this domain due to a fundamental data bottleneck. Specifically, ocean data are highly fragmented across disparate sources and inherently exhibit multi-modal, high-noise, and weakly labeled characteristics, lacking unified schemas and semantic alignment. Although Multimodal Large Language Models (MLLMs) have achieved remarkable success in general domains, their application to ocean science remains severely constrained by the absence of large-scale, well-aligned multimodal datasets tailored to marine environments. To bridge this gap, we introduce OceanPile, a large-scale multimodal corpus designed for ocean foundation models. It comprises three key components: OceanCorpus, a unified collection integrating sonar data, underwater imagery, marine science visuals, and scientific text from diverse authoritative sources; OceanInstruction, a high-quality instruction dataset synthesized via a novel pipeline guided by a hierarchical Ocean Concept Knowledge Graph; and OceanBenchmark, a manually curated evaluation benchmark for rigorous assessment. We establish a multi-stage quality control process to ensure scientific validity and alignment across modalities. Experimental validation demonstrates significant performance improvements for models trained on our data. All datasets are publicly released to advance the field of marine artificial intelligence and empower domain-specific MLLMs.
Background & Summary
OceanPile addresses marine AI’s data bottleneck by unifying fragmented, multimodal, noisy, and weakly labeled ocean data with semantic alignment. It provides a large-scale corpus, specialized processing pipeline, and evaluation benchmarks for domain-specific marine MLLMs.
- Data bottleneck: Ocean data are fragmented across literature, reports, and instruments, lacking unified schemas and semantic alignment while exhibiting multimodal, high-noise, and weakly labeled characteristics.
- OceanPile: OceanPile is a large-scale multimodal corpus that integrates dispersed oceanographic sources into an open-access resource for ocean foundation models.The corpus provides aligned data across sonar, imagery, and text modalities.
- Components: OceanPile comprises OceanCorpus, OceanInstruction, and OceanBenchmark as its three core components.
- Data processing: A specialized data-processing pipeline preserves scientific context and ensures semantic alignment across heterogeneous oceanographic sources for training domain-specific MLLMs.These processes maintain the scientific integrity and contextual richness of oceanographic information.
- Evaluation: Comprehensive evaluation benchmarks show that OceanPile significantly enhances MLLM performance on marine intelligence tasks.
Methods
The methods construct OceanPile through a unified multimodal corpus, standardized preprocessing, knowledge-graph-guided instruction synthesis, and expert-driven evaluation benchmark creation. These procedures target comprehensive, scientifically valid coverage of marine data and concepts.
- OceanCorpus: OCEANCORPUS integrates authoritative oceanographic data spanning textbooks, papers, marine web pages, sonar datasets, and underwater imagery.Its coverage includes chemical, biological, geological, and physical oceanography, alongside acoustic and optical underwater modalities.
- OceanCorpus: Field deployments use AUVs equipped with sonar and high-resolution optical cameras to capture synchronized data in diverse Zhoushan underwater scenes.The deployments address restricted object categories and artificial laboratory tank conditions in existing datasets.
- Preprocessing: Source documents are converted from structured formats or PDFs into clean text while preserving logical structure, images, tables, and hierarchy.The refined corpus removes low-quality and repetitive information while retaining technical accuracy and contextual richness.
- OceanInstruction: OCEANINSTRUCTION builds a hierarchical Ocean Concept Knowledge Graph from textbooks and expert-curated materials, then maps inputs to disciplines, refined subcategories, and external knowledge.GPT-4o extracts and merges candidate subcategories, while frequency filtering produces the refined concept structure used for multimodal instruction synthesis.
- OceanBenchmark: OCEANBENCHMARK contains separate textual and multimodal sub-benchmarks built through expert selection of authoritative content and professional multiple-choice question design.The textual benchmark targets text-only comprehension, whereas the multimodal benchmark targets multimodal reasoning.
Data Records
OceanPile is publicly released as three structured resources: OceanCorpus, OceanInstruction, and OceanBenchmark. Together, they provide multimodal marine data, text-only and multimodal instructions, and standardized textual and multimodal evaluation tasks.
- Release and format: OceanPile is publicly available on Hugging Face and contains the three resources for marine-oriented MLLM research and development.Benchmark instances include questions, correct answers, images for multimodal tasks, and detailed metadata, with standardized CSV-based evaluation.
- OceanCorpus: OceanCorpus integrates five categories spanning marine science texts, web pages, sonar databases, underwater imagery, and field-collected underwater data.It serves as the foundational multimodal data collection.
- OceanInstruction: OceanInstruction provides separate text-only and multimodal instruction datasets in CSV format, with question and answer fields plus image paths for multimodal instances.The multimodal version links each task to associated visual content.
- OceanBenchmark: OceanBenchmark evaluates marine-science capabilities through curated textual and multimodal tasks, including Ocean Science QA with 102 samples, Sonar VQA with 796, and Marine Organisms VQA with 472.The benchmark covers scientific question answering, sonar interpretation, and fine-grained marine-organism recognition.
Technical Validation
OceanPile’s technical validation combines multi-round human verification with evaluations of fine-tuned baseline models and closed-source MLLMs. Fine-tuning produces gains on textual and multimodal ocean benchmarks, including substantial improvements across all three multimodal sub-benchmarks.
- Human Verification: OCEANINSTRUCTION undergoes multi-round human verification by independent evaluators assessing factual accuracy, relevance, clarity, and marine-domain appropriateness.Each instance is independently scored by multiple evaluators after initial annotation.
- Model Performance Improvement: On Ocean Science QA, fine-tuning improves Qwen3-30B-A3B-Instruct from 25.49 to 26.47, exceeding GPT-5 (16.67) and GPT-4o (6.86).The fine-tuned model closely approaches Gemini-3-Flash (24.51).
- Model Performance Improvement: On multimodal benchmarks, Qwen3-VL-8B-Instruct improves from 21.21 to 29.29 on Ocean Science VQA, 8.04 to 19.97 on Sonar VQA, and 9.96 to 48.52 on Marine Organisms VQA.These gains span all three reported multimodal sub-benchmarks.