Source-linked AI summary
SonarLLM: A Native Sonar--Optical Multimodal Large Language Model for Underwater Perception
Cong Su, longxuan ma, Ling Dong, Guofeng Tang, Weijie Yin, Haohui Chen, Zhengtao Yu
TL;DR
Underwater optical sensing degrades with turbidity, while existing optical-centric MLLMs are poorly suited to sonar and adaptive sonar–optical reasoning. SonarLLM introduces native sonar representation, modality-specific enhancement, and reliability-aware hierarchical fusion, together with SonarBench’s controlled paired evaluation. It achieves strong sonar-only and fusion performance, with fusion benefits increasing as optical quality deteriorates.
Problem
Existing MLLMs rely predominantly on optical imagery and do not adequately model sonar’s geometry, artifacts, or observation-dependent reliability for open-ended language reasoning.
Method
SonarLLM uses an independent sonar encoder, modality-specific feature enhancement, quality-aware modality weighting, hierarchical interaction, and progressive training, evaluated with the paired SonarBench benchmark.
Results
72.0% sonar-only macro accuracy and 68.7% fusion accuracy exceed the strongest baselines by 34.4 and 25.1 points, while fusion-over-optical gains rise from 6.0 to 36.0 points as turbidity increases.
Takeaways & Limitations
The results support tailoring heterogeneous sensor representations to their observation processes and weighting modalities according to observation quality.
Takeaways & Limitations
The controlled protocol does not fully validate naturally occurring turbidity, motivating future evaluation on real paired data.
Abstract
from arXiv · showhide
Reliable underwater perception requires complementary sensing under variable visibility. Optical cameras capture appearance and semantics but degrade rapidly with turbidity, whereas imaging sonar preserves geometry while exhibiting distinct range-azimuth structure and acoustic artifacts. Existing MLLMs, built primarily on optical encoders, are therefore ill-suited to model sonar or adaptively exploit sonar-optical complementarity. We propose SonarLLM, a sonar-optical MLLM that treats sonar as a native perceptual modality. It combines a sonar-specific encoder, modality-specific physics-aware feature enhancement, and reliability-aware hierarchical fusion to align acoustic structure with optical semantics and dynamically adjust their contributions as sensing quality changes. We also introduce SonarBench, a paired benchmark that spans four tasks: recognition, counting, visual question answering, and captioning; and, across the benchmark, three input settings: sonar-only, optical-only, and fusion. By fixing the scene and sonar observation while varying optical degradation, SonarBench enables controlled measurement of cross-modal complementarity. SonarLLM achieves 72.0% macro accuracy across sonar-only recognition, counting, and VQA, outperforming the strongest baseline by 34.4 percentage points, and 68.7% under fusion, exceeding the best baseline by 25.1 points. For recognition and counting, the fusion-over-optical gain grows from 6.0 to 36.0 points as turbidity increases, indicating the increasing complementary value of sonar under controlled optical degradation. Together, these results show that robust heterogeneous perception depends not only on adding sonar, but on representing and weighting it according to its sensing characteristics.
1 Introduction
The paper introduces SonarLLM, a sonar–optical MLLM designed for modality-specific representation and reliability-aware fusion, alongside SonarBench for controlled complementarity evaluation. Experiments show strong sonar understanding and increasing fusion benefits as optical quality deteriorates.
- Underwater optical perception loses color, texture, contrast, and object boundaries with turbidity, whereas sonar preserves contours, spatial structure, and range information.
- Existing systems lack open-ended language reasoning for sonar and are poorly matched to sonar’s range–azimuth geometry, artifacts, and observation-dependent reliability.Fixed fusion can be vulnerable when either optical or acoustic evidence is unreliable.
- SonarLLM combines an independent sonar encoder, modality-specific feature enhancement, quality-aware weighting, hierarchical fusion, and progressive training.Its acoustic enhancement targets reverberation and range attenuation, while optical enhancement targets scattering-related corruption.
- SonarBench evaluates recognition, counting, VQA, and captioning across sonar-only, optical-only, and fusion settings while varying optical quality for fixed scenes and sonar observations.This protocol separates unimodal modeling gains from complementary sonar evidence as optical reliability deteriorates.
- 72.0% macro accuracy on sonar-only recognition, counting, and VQA beats the strongest baseline by 34.4 points, while fusion reaches 68.7%, 25.1 points above the best baseline.For recognition and counting, fusion-over-optical gains increase from 6.0 points in clear conditions to 36.0 points under heavy turbidity.
2 Related Work
Prior underwater vision-language and sonar research provides domain data, priors, and task-specific fusion, but does not address sonar-native open-ended reasoning under paired optical degradation. Existing benchmarks likewise cannot isolate unimodal improvements from complementary sonar evidence.
- MarineGPT, AquaticCLIP, NAUTILUS, and OceanGPT extend vision-language learning to marine settings through domain data, contrastive pretraining, enhancement, or ocean knowledge.
- OceanPile and OceanGym provide ocean multimodal data and embodied evaluation, but do not study sonar-native open-ended reasoning under paired optical degradation.
- Existing sonar research and paired datasets primarily support detection, classification, segmentation, tracking, or other specific discriminative tasks rather than language-level interaction.
- General-purpose MLLMs use projection, Q-Formers, cross-layer attention, or hierarchical visual features, while some fusion methods adjust modality contributions according to sensor quality.The passage states that these approaches generally assume visually homogeneous inputs.
- Existing evaluation resources are divided by modality or task and cannot isolate stronger unimodal modeling from genuinely complementary sonar evidence.Optical enhancement, sonar perception, paired-sensor tasks, and optical vision-language benchmarks remain separate.
3 Method
SonarLLM combines sonar-native representation, modality-specific enhancement, reliability-aware fusion, and progressive training to support heterogeneous sonar–optical language understanding.
- Sonar-Native Visual Representation: SonarLLM preserves Qwen3-VL-8B’s optical encoder and adds an independent sonar encoder, treating sonar as a native perceptual modality.The sonar pathway uses a Polar-aware Sonar Vision Transformer initialized from the Qwen3-VL visual tower.
- Sonar-Native Visual Representation: The sonar encoder adapts transferred visual features with a multi-scale Sonar Stem and explicit range–azimuth positional encoding.The stem captures echoes, boundaries, and acoustic shadows at different receptive fields, while positional functions encode physical range and azimuth.
- Modality-Specific Feature Enhancement: Separate Optical-VFE and Acoustic-VFE modules correct modality-specific feature corruption without attempting raw physical image-formation inversion.Optical-VFE targets scattering-related corruption, whereas Acoustic-VFE models reverberation and range-dependent attenuation.
- Reliability-Aware Hierarchical Fusion: AGFM estimates token quality and modality-level reliability, then reweights optical and sonar streams while preserving separate representations.Single-modality inputs reduce AGFM to an identity mapping, and the reweighted streams are injected through dual-stream DeepStack at multiple language-model layers.
- Progressive Training Strategy: Progressive training establishes sonar-domain representations, acoustic semantics, cross-modal alignment, reliability learning, and instruction following in four stages.The schedule begins with masked reconstruction on unlabeled sonar images, adds category supervision, trains alignment and gates, and ends with multimodal instruction tuning.
4 Experiments
Experiments evaluate SonarLLM’s sonar-native perception, controlled sonar–optical complementarity, reliability-aware fusion, representation formation, ablations, and computational cost. Results show strong sonar-only and fusion performance, increasing sonar value under turbidity, component-specific contributions, and measurable overhead.
- Main Results: 72.0% sonar-only macro accuracy exceeds the strongest baseline by 34.4 points across recognition, counting, and VQA.Performance is 64.7% for recognition, 84.7% for counting, and 66.5% for VQA; caption GOOD rate reaches 33.3% versus 12.0%.
- Main Results: 68.7% fusion accuracy exceeds the strongest baseline by 25.1 points, while fusion recognition remains between 76.7% and 83.3% across degradation levels.Relative to the same Stage-IV and language-side LoRA control, SonarLLM improves sonar, optical, and fusion accuracy by 34.4, 14.0, and 36.5 points.
- Controlled Degradation: The fusion-over-optical gain for recognition and counting expands from 6.0 to 36.0 points as conditions move from clear to heavy turbidity.Fusion changes little from clear to heavy conditions, while optical performance declines substantially; recognition fusion also exceeds the stronger individual sensor by 6.0, 12.0, and 16.0 points.
- Reliability-Aware Fusion: Sonar weight rises with degradation, from 0.491 to 0.795, with Spearman correlation 𝜌= 0.749 and non-decreasing weights in 95.3% of scenes.At the same checkpoint, AGFM’s advantage over equal weighting grows from 0.0 to 2.7 points, and the clear-to-heavy decline is 0.7 points versus 3.4 points.
- Ablation and Representation Analysis: Representation and ablation analyses attribute gains to sonar geometry, modality-specific enhancement, hierarchical interaction, and quality-aware weighting.Removing Polar PE reduces fusion/sonar-macro accuracy by 13.5/10.1 points; removing DeepStack causes a 6.2-point sonar-macro loss, while AGFM reduces degradation sensitivity.
- Computational Cost: 17.4% more parameters, 22.0% more peak memory, 78.1% higher prefill latency, and 45.1% lower decoding throughput quantify the cost of native sonar processing.Most added capacity is inherited from a pretrained and frozen sonar tower and DINOv2; 25.1M non-LoRA adaptation and fusion parameters are randomly initialized.
5 Conclusion
SonarLLM supports robust sonar–optical reasoning by tailoring representations to each modality and conditioning fusion on observation quality. Its controlled evaluation shows sonar becomes increasingly valuable as optical quality deteriorates, while the benchmark protocol does not fully validate naturally occurring turbidity.
- 72.0% sonar-only macro accuracy and 68.7% fusion accuracy exceed the strongest baselines by 34.4 and 25.1 points, respectively.
- The fusion-over-optical gain rises from +6.0 to +36.0 points as optical quality deteriorates.
- Robust sonar–optical reasoning requires modality-tailored representations and fusion conditioned on observation quality.
- The controlled protocol does not fully validate naturally occurring turbidity, motivating evaluation on real paired data, task-aware routing, and temporal reasoning.