PaperlayerView exact revision on arXiv

Source-linked AI summary

UA-Bench: Benchmarking Large Audio-Language Models on Underwater Acoustic Recognition

Zhe Sun, Liang An, Jiayu Yao, Ronglai Qian, Jianing Cai, Ziwei Huang, Qing Fan, Zhiyuan Yang, Yujun Cai, Hongli Cao, Yiwei Wang, Xiaoyan Wang

arXiv:2610.16942v1cs.SDcs.AI

TL;DR

Underwater acoustic recognition matters for maritime applications, but LALMs have not been systematically evaluated on these signals, and creating controlled data is challenging. UA-Bench combines traceable synthetic sources and propagation with hierarchical labels and diagnostic evaluation; nine open models show low leaf accuracy and limited waveform gains at that level.

  • Problem

    LALMs have not been systematically evaluated on underwater signals, where recognition involves distinct signal structures and propagation effects and annotated recordings are costly.

    Source: §1 · pp. 1–2

  • Method

    UA-Bench pairs parameterized underwater sources with modeled propagation and traceable metadata, then evaluates hierarchical recognition with a gated protocol and paired silent-audio control.

    Source: §1 · p. 2, §3.1 · p. 6, §4.5 · p. 18

  • Results

    2.0%–13.1% leaf accuracy across nine open LALMs compares with a 7.7% random reference, and no model leads across all three signal families.

    Source: §1 · p. 3

  • Takeaways & Limitations

    UA-Bench provides a common basis for studying hierarchical recognition in relation to signal families, decision stages, generation metadata, and explanation vocabulary.

    Source: §5 · p. 18

    The benchmark represents a synthetic distribution, so field recordings are needed to measure transfer beyond its source ranges, propagation settings, evaluation selection, questions, and decoder.

    Source: §5 · p. 18

Abstract

from arXiv · show

Large audio-language models (LALMs) are fast becoming a general interface to sound, yet how far their broad pretraining carries into specialized domains remains largely open. Vessel monitoring, maritime safety, and ocean sensing all depend on recognizing underwater signals, but to our knowledge LALMs have not been systematically evaluated on them. Annotated recordings are costly to collect, and a faithful synthetic alternative has to bring together many interacting factors: the structure of detection pulses, communication waveforms, and ship-radiated noise, and the sound-speed profile, geometry, and multipath propagation through which each reaches the receiver. We address this by proposing \textbf{UA-Bench}, to our knowledge the first large-scale benchmark for evaluating LALMs on underwater acoustic signal recognition. It comprises 375,051 audio records that couple parameterized sources with modeled propagation, so that every record can be traced to the conditions that produced it. The records span these three signal families, and each carries a three-level label hierarchy (13 leaf classes), a templated multi-turn dialogue, and source and channel metadata. We evaluate on a balanced 2,600-record subset with a gated coarse-to-fine protocol that locates each error at source routing, fine-grained classification, or output format, and add a paired silent-audio control to test whether the waveform is actually used. Nine open LALMs, evaluated without task-specific adaptation, reach only 2.0\%--13.1\% leaf accuracy, against a random reference of 7.7\%. The waveform helps source routing far more than leaf recognition (for one model, +29.5 versus +2.3 points over silence), and just four models gain reliably at the leaf level, by at most 4.7 points. UA-Bench thus gives a reference point for these models, and its released training partition supports follow-up adaptation work.

1 Introduction

UA-Bench addresses the limited evaluation of LALMs on underwater signals, where scarce labels and complex source and propagation factors make controlled recognition and diagnosis difficult. Its benchmark and gated evaluation reveal low fine-grained accuracy and limited waveform gains at the leaf level.

1 Introduction

  • Underwater signals differ in their defining structure, while sound-speed profiles, source–receiver geometry, and multipath reshape received waveforms and annotated recordings are costly to acquire.These conditions make controlled synthetic data valuable for tracing errors to source or propagation factors.Source: §1 · p. 2
  • Existing audio benchmarks omit engineered underwater signals, while most underwater datasets support supervised closed-set vessel recognition and WoW-Bench focuses on marine mammal vocalizations.UA-Bench brings engineered signals, known generation conditions, and a question-answering interface together.Source: §1 · p. 2
  • UA-Bench contains 375,051 traceable clips across 13 leaf classes, paired with templated dialogues and source-grouped partitions, plus a balanced 2,600-clip evaluation subset.Parameterized sources are propagated with BELLHOP under varied sound-speed profiles and geometry; source and channel conditions are retained for each record.Source: §1 · p. 2
  • A gated coarse-to-fine dialogue assigns errors to source routing, fine-grained classification, or output format, while paired equal-duration silence tests reliance on the waveform.The benchmark also compares performance across acoustic conditions and checks whether explanations agree with predicted labels.Source: §1 · p. 3, Figure 2 · p. 3
  • 2.0%–13.1% leaf accuracy across nine open LALMs compares with a 7.7% random reference, and no model leads on all three signal families.The models also fail at different stages, including answer format, source routing, and concentration on a few classes.Source: §1 · p. 3

2 Related Work

Prior work spans broad audio-language benchmarks, specialized audio pretraining, and underwater recognition resources, but these strands leave controlled LALM evaluation on engineered underwater signals largely open.

2 Related Work

  • UA-Bench combines three underwater signal families and 13 leaf classes in 375,051 records, with controlled source and propagation metadata and a fixed 2,600-record evaluation suite.The related-work comparison identifies it as the first large-scale, multi-category LALM benchmark for underwater acoustic signal recognition.Source: §2 · p. 4

2.1 Audio Understanding Benchmarks

  • Audio benchmarks have broadened from speech, music, and environmental sound toward reasoning and marine mammal vocalizations, while engineered underwater signals and propagation variation remain largely outside their scope.Examples include Dynamic-SUPERB Phase-2, GAMA, and WoW-Bench.Source: §2.1 · p. 4

2.2 General Audio Pretraining in Specialized Acoustics

  • Broad audio pretraining has supported zero-shot animal recognition and classifiers for respiratory and heart sounds, but noisy-task performance can decline; most specialized-domain studies use task-specific adaptation.UA-Bench instead keeps the model fixed while varying source family and propagation conditions under controlled settings.Source: §2.2 · pp. 4–5

2.3 Underwater Acoustic Recognition, Data, and Simulation

  • Underwater datasets preserve real environmental complexity but couple source and propagation, whereas simulators provide forward models without an evaluation inventory, labels, partitions, dialogue task, or diagnostic protocol.UA-Bench connects parameterized synthesis and BELLHOP propagation with validated waveforms, retained metadata, and source-grouped partitions.Source: §2.3 · p. 5

3 UA-Bench

UA-Bench combines controlled underwater signal synthesis, structured dialogue annotations, source-grouped partitions, and a gated evaluation protocol to assess where large audio-language models fail. Its evaluation subset is class-balanced, but its acoustic conditions differ from those of the full test partition.

3 UA-Bench

  • The benchmark links parameterized sources and modeled propagation with a three-level label hierarchy, templated dialogue, and source and channel metadata.Each record pairs a retained clip with its labels and dialogue; the evaluation protocol tests whether errors arise in coarse routing, fine classification, or output format.Source: §3 · p. 5, §3.1 · p. 6, §3.3 · p. 8

3.1 Task Scope and Label Hierarchy

  • Thirteen leaf classes are organized by source category and signal family, with Active covering detection pulses and communication waveforms and Passive covering ship-radiated noise.Turn 1 selects Active or Passive; Turn 2 selects a complete path within that branch, with eight Active and five Passive options.Source: Table 2 · p. 5, §3.1 · p. 6

3.2 Controlled Source and Channel Synthesis

  • Each clip is built from a class-conditioned source and sampled propagation channel, then peak-normalized; metadata retain source and channel conditions.BELLHOP simulates propagation using sampled sound-speed profiles and geometry, and retained waveforms preserve relative temporal and spectral structure rather than absolute propagation loss.Source: §3.2 · pp. 6–8

3.3 QA Annotation Construction

  • Each clip is paired with three-turn questions and reference answers, while evaluation models receive audio, questions, and prior answers but not labels or metadata.The first two turns assess classification; the third requests a brief acoustic explanation assembled from source metadata.Source: §3.3 · p. 8

3.4 Dataset Partitions and Evaluation Selection

  • UA-Bench-Eval contains 2,600 records, with 200 sampled per leaf class from source-grouped test partitions; its selected acoustic conditions differ from the full test partition.Selected active-signal channel gains and ship tonal-to-continuum ratios are concentrated at larger values, active frequencies shift upward, and selected pulses skew shorter in duration.Source: §3.4 · pp. 8–9, Figure 5 · p. 10

3.5 Gated Evaluation Protocol and Classification Metrics

  • The gated protocol assigns errors to source routing, fine-grained classification, or output format, and reports stage-wise accuracy and conditional accuracy.An incorrect or invalid Turn 1 answer ends the dialogue; a correct route opens the relevant branch for Turn 2, whose complete-path option determines both L2 and L3.Source: §3.5 · pp. 10–11

3.6 Diagnostic Analyses

  • Diagnostic analyses pair each record with equal-duration silence, stratify recall by generation metadata, and assess whether explanation terms align with predicted leaves.Silent-audio comparisons report original-minus-silent differences with source-cluster bootstrap intervals; explanation contradiction rate combines Concept Confusion and Term Stacking.Source: §3.6 · p. 11

4 Experiments

Across nine strict runs, L3 accuracy is low and model strengths vary across source families. Metadata-stratified results show no shared monotonic trend, while paired controls find waveform effects that differ by model and hierarchy level.

4.1 Experimental Setup

  • The experiments evaluate nine released instruction-following checkpoints without task examples on a fixed 2,600-record suite, using each model’s native audio processor and chat template.The authors caution that comparisons are checkpoint-configuration results rather than architecture-only effects.Source: §4 · p. 12, §4.1 · p. 12

4.2 Overall and Family Performance

  • 2.0%–13.1%: L3 accuracy spans this range across nine strict runs; Qwen2.5-Omni leads accuracy, while MiDashengLM leads Macro F1 at 0.083.Qwen2.5-Omni’s L3 accuracy is 5.4 percentage points above the gated random reference after rounding; its Macro F1 is 0.082.Source: Table 4 · p. 12, §4.2 · p. 12
  • Voxtral-Small leads on detection pulses, Gemma-4-E2B-IT on communication waveforms, and Qwen2.5-Omni on ship-radiated noise.No model maintains its lead across all three families; strict-run predictions are also concentrated in particular classes.Source: Table 5 · p. 13, §4.2 · pp. 13–14
  • Gemma-4-12B’s dominant error is 1,156 invalid-format Turn 1 refusals, while Audio-Flamingo-Next has 841 invalid-format Turn 2 answers.Qwen2.5-Omni reaches Turn 2 on 1,886 examples without an invalid-format decision and makes 341 correct terminal predictions; Voxtral-Mini has no invalid-format answers but 1,378 valid, incorrect terminal decisions.Source: Table 6 · p. 14, Figure 7 · p. 14, §4.2 · p. 15

4.3 Associations with Generation Metadata

  • Metadata strata show no shared monotonic recall pattern across checkpoints; Active endpoint differences range from +3.2 percentage points for Gemma-4-E2B to −3.0 for Voxtral-Mini.Qwen2-Audio has zero Active recall, while its ship-noise recall rises from 16.4% in the low stratum to 21.2% in the high stratum.Source: Table 7 · p. 15, §4.3 · p. 15, Figure 8 · p. 16

4.4 Lexical Explanation Analysis

  • Gemma-4-12B has the highest aligned lexical rate at 87.6%, but this is conditional on 717 eligible Turn 3 responses.Voxtral-Mini follows at 62.6%; Voxtral-Small has the lowest vague rate at 5.6%, and MiDashengLM the lowest concept-confusion rate at 1.6%.Source: Table 8 · p. 16, §4.4 · p. 16

4.5 Silent-Audio Control

  • Waveform gains are larger at L1 routing than L3 recognition; Audio-Flamingo-Next gains +29.54 percentage points at L1 versus +2.35 at L3.Figure 9 compares original audio with duration-matched silence for the same checkpoints.Source: Figure 9 · p. 17, §4.5 · p. 17, Table 9 · p. 17
  • Four models have positive L3 accuracy differences whose cluster-bootstrap intervals exclude zero; Qwen2.5-Omni’s difference is largest at 4.7 percentage points.These waveform-effect conclusions are specific to the evaluated configurations and should not be generalized beyond the paired control.Source: Table 9 · p. 17, §4.5 · pp. 17–18

5 Discussion and Conclusion

UA-Bench combines hierarchical recognition, source and propagation metadata, and a gated evaluation protocol to measure source routing and leaf classification. Across evaluated models, fine-grained accuracy is limited, and no model leads consistently across signal families.

5 Discussion and Conclusion

  • UA-Bench connects thirteen leaf classes and source-grouped partitions with a fixed evaluation suite and a gated protocol measuring source routing and leaf classification.Predictions are related to signal families, decision stages, generation metadata, and explanation vocabulary.Source: §5 · p. 18
  • L3 accuracy ranges from 2.0% to 13.1% across nine strict-decoder runs, and no evaluated model leads consistently across all three signal families.MiDashengLM has the highest Macro F1 at 0.083, narrowly ahead of Qwen2.5-Omni at 0.082; leaders differ by family.Source: §5 · p. 18
  • Four models show positive L3 differences over silence with cluster-bootstrap intervals excluding zero, while Voxtral-Mini performs better on silence in this diagnostic comparison.The outcomes describe behavior under this protocol, not whether audio is generally useful.Source: §5 · p. 18
  • The benchmark uses a synthetic distribution; field recordings are needed to measure transfer beyond its source ranges, propagation settings, evaluation selection, questions, and decoder.Future work can expand environmental diversity, add independently validated labels, and compare decisions under controlled audio changes.Source: §5 · p. 18

A Source Synthesis Settings

The source generators create active pulses and communication waveforms alongside ship-radiated noise, with settings varied across source families. Figures show representative evaluation inputs and ship-noise spectral summaries.

A Source Synthesis Settings

  • The active generators produce 160,000 source waveforms at 16 kHz, with occupied frequencies from 100 to 7900 Hz.The groups include 24,000 CW waveforms, 28,000 for each chirp class, and 16,000 for each communication modulation class.Source: A Source Synthesis Settings · p. 21
  • Pulse duration depends on the selected subband, communication duration on symbol count and rate, and OFDM settings also vary the cyclic prefix and subcarrier count.Table 10 summarizes the active-signal synthesis rules by source family.Source: A Source Synthesis Settings · p. 21, Table 10 · p. 21
  • Each of five vessel classes has 16,000 source waveforms combining a shaped random continuum with discrete mechanical tones.The continuum is flat below 300 Hz and falls by 6 dB per octave above that frequency; each source contains three to ten tones, with one to three attempted shaft or blade harmonics.Source: A Source Synthesis Settings · p. 21
  • Blade count, shaft frequency, spectrum level, and principal modulation coefficient are sampled from specified ranges; the underwater vehicle represents a generic propeller-driven platform.It is presented to evaluated models under the taxonomy name Underwater vehicle.Source: A Source Synthesis Settings · p. 21
  • Figure 10 displays representative time-frequency structures for three pulse types and five communication waveforms, while Figure 11 shows ship-noise spectra and class-wise spectral-centroid distributions.Figure 11’s spectra are individual inputs, not class-average templates.Source: Figure 10 · p. 22, A Source Synthesis Settings · p. 22, Figure 11 · p. 23

B Propagation Response Construction

Propagation responses are constructed from selected sound-speed profiles and perturbed source-receiver geometry, then applied to active waveforms and ship-radiated noise. The pipeline validates the resulting waveforms before retaining audio and metadata.

B Propagation Response Construction

  • Each propagation realization selects a sound-speed profile and independently perturbs source depth, receiver depth, and range by up to 10%, keeping depths within the water column.If a run has no arrivals and uses fewer than 8000 beams, it is repeated once with a larger beam count under the stated rule.Source: B Propagation Response Construction · p. 23
  • The arrival parser accepts finite, nonnegative delays and finite, positive amplitudes, retaining at most twenty arrivals by amplitude while preserving the earliest.That earliest retained arrival defines the common delay reference for usable ship frequency responses.Source: B Propagation Response Construction · p. 23
  • Active waveforms are convolved with the response at their center or carrier frequency, while ship-noise responses between adjacent frequencies are linearly interpolated.Outside the available frequency range the nearest endpoint is used; the inverse transform is cropped to the source frame count and peak-scaled to 0.95.Source: B Propagation Response Construction · p. 23, Table 12 · p. 24
  • A realization is retained only with a usable response and a finite, nonzero output waveform; audio and metadata are written after these checks succeed.Active realizations require one usable response, while ship realizations require at least one usable representative frequency.Source: B Propagation Response Construction · p. 24

C Metadata, Records, and Dataset Selection

The records preserve source and channel information alongside labels and identifiers. Source groups remain together across partitions, and the evaluation subset is selected from the test partition using per-class rankings.

C Metadata, Records, and Dataset Selection

  • Metadata records the active impulse-response energy gain before output scaling and a ship-noise SNR statistic after mixing, with the latter reconstructed from saved tonal components.The ship statistic differs from the local ratio used to set each individual tone.Source: C Metadata, Records, and Dataset… · p. 24
  • The annotation stage copies source and channel metadata into each record and adds the canonical label path and source identity.The channel suffix is removed from the record identifier to recover source identity; Table 13 summarizes the released fields and selection settings.Source: C Metadata, Records, and Dataset… · p. 24, Table 13 · p. 24
  • Source groups are split by leaf class into nominal 70/15/15 training, validation, and test partitions, and all propagation realizations of a source inherit its partition.UA-Bench-Eval samples 200 records without replacement from the first 400 ranked test records in each leaf-class pool, using seed 42.Source: C Metadata, Records, and Dataset… · pp. 24–25

D Question and Reference Construction

D Question and Reference Construction defines a stored, templated dialogue that moves from source category to a branch-specific fine-grained class, then requests an acoustic explanation. Fixed option mappings and answer-format instructions make the classification steps evaluable.

D Question and Reference Construction

  • Question wording is selected once during annotation and stored with each record, while evaluation reuses those questions and supplies the waveform and earlier answers.Reference answers and metadata are not sent to the model.Source: D Question and Reference Construction · pp. 25, 27

TURN 1 SOURCE CATEGORY

  • Turn 1 asks whether the sound is an intentional transmission or source-radiated noise, using the same two options and answer instruction across templates.The question pool contains eight Turn 1 templates, and the selected wording is stored in each record for evaluation reuse.Source: D Question and Reference Construction · p. 25

TURN 2 SIGNAL FAMILY AND SPECIFIC CLASS

D Question and Reference Construction

  • The decoder accepts only a single ASCII option letter, treats letter case equivalently, and stops the dialogue after an invalid-format or incorrect Turn 1 answer.A valid Turn 2 letter resolves a unique L2–L3 path after correct Turn 1 routing.Source: D Question and Reference Construction · p. 26
  • The Turn 3 prompt requests observed acoustic cues supporting the previous classification and instructs models to state explicitly when evidence is insufficient.Reference explanations combine source, signal-family, and acoustic descriptors; descriptor selection varies by waveform family and vessel-noise characteristics.Source: D Question and Reference Construction · pp. 26–27

TURN 3 AUXILIARY EXPLANATION

The dialogue’s final turn asks for an acoustic explanation after source routing and complete-path classification. The 2FSK example illustrates how the evaluator compares the response with terms supporting the predicted leaf.

TURN 3 AUXILIARY EXPLANATION

  • The explanation prompt asks for identifiable acoustic cues supporting the prior choice and prohibits listing alternatives or inventing evidence when cues are insufficient.In the illustration, two discrete frequency states and symbol-wise frequency switching are supporting terms, and the response is labeled Aligned.Source: TURN 3 AUXILIARY EXPLANATION · p. 28
  • Classification labels remain unchanged during the auxiliary explanation turn.Source: TURN 3 AUXILIARY EXPLANATION · p. 28
  • The 2FSK example follows source routing, complete-path classification, and an auxiliary explanation, with audio retained across turns.The evaluator parses the first two answers before comparing the explanation with the predicted 2FSK inventory.Source: Figure 12 · p. 28, TURN 3 AUXILIARY EXPLANATION · p. 28

E Lexical Explanation Analysis

Lexical explanation analysis matches Turn 3 wording against terms compatible with the predicted leaf and the remaining inventory. Its categories partition executed responses, with special handling for invalid-format Turn 2 predictions.

E Lexical Explanation Analysis

  • The term inventory is organized by predicted leaf, with shared cues assigned to every compatible leaf; vessel leaves share vessel terms and add their class name.Matching is case-insensitive, uses letter and digit boundaries, and retains the longest non-overlapping candidate phrases.Source: E Lexical Explanation Analysis · p. 29, Table 15 · p. 29
  • The first four lexical categories partition executed responses, while an invalid-format Turn 2 prediction has no supporting terms and uses the full inventory as its comparison set.Such a response is Concept Confusion if any inventory term appears and Vague otherwise; an empty explanation is also Vague.Source: E Lexical Explanation Analysis · p. 30

F Classification Metrics and Random References

The section specifies classification metrics, random references, and uncertainty estimation for the paired silent-audio control. Under the strict decoder, valid fine-grained correctness also implies correct source and signal-family labels.

F Classification Metrics and Random References

  • L3 Macro F1 is the unweighted mean of the thirteen leaf-class F1 values, and invalid-format or cascade predictions count as false negatives for true classes.Those predictions also appear as separate outcomes in the confusion matrix.Source: F Classification Metrics and Random… · p. 30
  • A valid Turn 2 option that is correct at L3 necessarily has correct source and signal-family labels, so L3 accuracy equals complete-path accuracy.Source: F Classification Metrics and Random… · p. 30
  • The random reference chooses either source category with probability 1/2; on the active branch, three of eight options are detection pulses and five are communication signals.The passive branch contains one signal family.Source: F Classification Metrics and Random… · p. 30
  • 1/13 is the stated random leaf reference, derived from L1 accuracy of 1/2 and conditional references of 37/52 for L2|L1 and 8/37 for L3|L2.Source: F Classification Metrics and Random… · p. 30
  • The silent-audio comparison reports 95% intervals for original and silent accuracy, their difference, and corresponding Macro F1 quantities using a source-identity cluster bootstrap.Channel variants are resampled together; the evaluator uses 10,000 replicates by default.Source: F Classification Metrics and Random… · p. 30

G Inference and Reproducible Evaluation

Inference uses each backend’s native processor and chat template, with a shared evaluation workflow and fixed evaluation records and thresholds. The protocol documents model settings and validates complete, consistent coverage before aggregating results.

G Inference and Reproducible Evaluation

Related papers

Loading 2610.16942v1…