PaperlayerView exact revision on arXiv

Source-linked AI summary

Audiovisual diarization of overlapping click trains in sperm whale (Physeter macrocephalus) vocal sparring using a three-hydrophone array

Lara Berkenbaum, Hervé Glotin, François Sarano, Walter M X Zimmer, Véronique Sarano, Olivier Adam, Pascale Giraudet

arXiv:2609.12593v1q-bio.QMeess.SP

TL;DR

Overlapping surface click trains are difficult to attribute to individual sperm whales, especially when similar-sized animals have indistinguishable IPIs. The paper combines spectro-temporal tracking and spatial localization with optical validation, assigning 11 intertwined click trains totaling 882 clicks to two immature males. The framework also shows that spatial errors can affect particular temporally entangled segments while overall spatial identity remains resolvable.

  • Problem

    Similar-sized immature males can have near-identical IPIs, making overlapping emissions acoustically indistinguishable and individual attribution difficult.

    Source: §I · p. 3

  • Method

    The workflow combines spectro-temporal tracking, TDOA spatial localization, and optical validation to assign click trains to identified individuals.

    Source: E. Acoustic spatial localization and… · p. 9, §V · p. 25

  • Results

    11 distinct, intertwined click trains totaling 882 clicks were assigned to two immature males engaging in vocal sparring.

    Source: §V · p. 25

  • Takeaways & Limitations

    Visual concordance provided ground truth that resolved ambiguities where purely acoustic TDOA estimates failed, supporting individual attribution during close-range interactions.

    Source: §V · p. 25

    Correlation phantoms induce systematic horizontal displacements during specific temporally entangled segments, although overall spatial identity remains resolvable.

    Source: B. Physical and algorithmic origins of… · p. 23

Abstract

from arXiv · show

The individual attribution of sperm whale (Physeter macrocephalus) vocalizations during surface interactions constitutes a methodological challenge due to acoustic overlaps, multipath propagation, body shadowing, and indistinguishable inter-pulse intervals among similar-sized individuals. A multimodal audio-visual workflow is presented to deinterleave and attribute uncharacterized click trains produced by immature males during ``vocal sparring'' using a portable three-hydrophone array coupled with synchronized video. The approach combines spectro-temporal tracking, based on inter-click interval dynamics and the Constant-Q Transform, with spatial time-difference-of-arrival modeling projected onto the image plane for optical validation. Analysis of 2,655 manually validated clicks shows that purely acoustic clustering diarization errors remain low, peaking at 25.79% only during extreme temporal superpositions. Integrating the optical modality resolves residual spatial indeterminacies; although near-planar array geometry induces vertical ambiguities, the system achieves up to 100% horizontal visual concordance for primary emitters. Crucially, this framework successfully reconstructed and assigned 11 distinct, intertwined click trains totaling 882 clicks to specific focal individuals despite near-field tactile constraints. Because ethological descriptions remain incomplete without identifying the emitter, explicitly correlating these emissions with physical kinematics provides the fine-scale resolution required to define this socio-acoustic behavior.

CIAN-RR-20260810

Continuous click trains during close-range socialization are difficult to attribute, particularly when similar-sized whales overlap. The paper proposes combining acoustic tracking and localization with optical validation to identify emitters.

I. INTRODUCTION

  • Continuous surface click trains do not fit the familiar categories of echolocation clicks and social codas, and overlap makes their individual attribution a major methodological bottleneck.The challenge is especially consequential for immature males, whose vocal ontogeny remains unmapped and who are morphologically indistinguishable from adult females.Source: §I · p. 3
  • Compact-array TDOA, surface multipath, irregular click timing, and changing signal orientation limit standard acoustic separation and localization during near-surface interactions.Same-sized individuals can also have indistinguishable IPIs, so visual identification and spatial concordance are needed to assign deinterleaved clicks to specific emitters.Source: §I · pp. 3–4
  • Macro-scale systems lack sub-meter precision and optical ground truth, biologging is invasive and unsuitable for multiple untagged animals, and coupled camera-hydrophone methods face sperm-whale-specific challenges.Those challenges include biosonar forward-directivity and surface multipath interference.Source: §I · p. 4
  • The proposed workflow combines CQT-based spectro-temporal tracking, spatial localization, and optical validation to assign emitters and evaluate acoustic diarization and visual concordance.Using vocal sparring as a proof of concept, the authors aim to test a non-invasive workflow with a mobile three-hydrophone array.Source: §I · p. 4

II. MATERIALS AND METHODS

The study examines vocal sparring between identified immature males using synchronized video and a compact three-hydrophone array. Its acquisition, scene selection, and preprocessing steps support acoustic clustering alongside spatial localization.

A. Study context and subjects

  • The study treats vocal sparring as non-aggressive dyadic contact with sustained overlapping click trains and analyzes interactions between two five-year-old immature males, Ali and Daren.The whales were identified using the local population photo-identification catalog.Source: A. Study context and subjects · pp. 4–5

B. Audio-visual acquisition system: OPALE array

A. Study context and subjects

  • The authors retained nine focal scenes from 12 interaction scenes after excluding three for field-of-view exits or incomplete click detection.Raw ICI coefficient of variation was used to quantify apparent scene complexity before deinterleaving.Source: B. Audio-visual acquisition system:… · p. 6

C. Acoustic pre-processing and click detection

  • Acoustic preprocessing calibrated channels, high-pass filtered signals, and used TKEO-based impulse detection with permissive thresholds for tracking and strict thresholds for TDOA.Detections were manually validated, and adaptive envelope tracking excluded distant biological choruses.Source: C. Acoustic pre-processing and click… · pp. 6–7
  • Scenes with ICI CV ≥0.5 were classified as highly overlapping, while separate clustering and localization chains used distinct signal conditioning and sensor inputs.Clustering used one reference hydrophone for spectral signatures; localization used all three hydrophones to estimate click origins.Source: C. Acoustic pre-processing and click… · p. 7

D. Spectro-temporal tracking and acoustic clustering

The clustering pipeline conditions a reference-hydrophone signal to reveal spectral structure, then combines CQT signatures with rhythmic constraints to reconstruct overlapping click trains. It also fuses fragments interrupted by temporal superposition.

D. Spectro-temporal tracking and acoustic clustering

  • A first-order derivative conditions the reference-hydrophone waveform before spectral extraction, flattening the 1/f noise profile and relatively enhancing high-frequency transients without removing low-frequency content.The +6 dB/octave pre-emphasis was calibrated to the SQ26 hardware profile.Source: D. Spectro-temporal tracking and… · pp. 7–8
  • CQT provides logarithmic frequency bins and adaptive dynamic resolution, while initial TKEO detections supply absolute timing for tracking.The CQT preserves high spectral resolution at lower frequencies to map environmental interference patterns.Source: D. Spectro-temporal tracking and… · p. 8
  • The CQT spans 1.2–80 kHz to retain low-frequency structure and geometry-dependent spectral content, and each click is aligned to its energy peak before signature extraction.Standardized time-frequency patches mitigate absolute gain effects associated with propagation distance.Source: D. Spectro-temporal tracking and… · p. 8
  • Tracking assigns clicks using spectral similarity to both an emitter’s evolving average profile and its last validated click, with short-term ICI inertia constraining timing.An a posteriori fusion phase reconnects fragments interrupted by temporal superposition and checks their global rhythmic consistency and spectral similarity.Source: D. Spectro-temporal tracking and… · pp. 8–9

E. Acoustic spatial localization and directional estimation

The section estimates click-train directions from inter-hydrophone delays, then aligns those estimates with synchronized video for emitter validation. Its metrics assess clustering assignments and optical projection accuracy, with important interpretive bounds.

E. Acoustic spatial localization and directional estimation

  • Pairwise TDOA estimates use frequency-domain cross-correlation and quadratic peak interpolation, followed by ≤10 cm loop-closure and spatial-residual filters.The temporal delays are converted to apparent distances using a local sound speed of 1541 m/s.Source: E. Acoustic spatial localization and… · p. 9
  • The planar sub-array yields only tangential direction directly; the missing normal component is reconstructed from unit norm, with the positive root selecting the frontal half-space.Array-frame shadowing and SQ26 directivity loss above 10 kHz support this postulate; visual target intersection retrospectively excludes symmetric ghost locations.Source: E. Acoustic spatial localization and… · pp. 9–10
  • After camera-frame angle correction, KDE modes characterize each train’s dominant emission axis and spatial concentration, checking whether acoustic clusters have separable directions.Source: E. Acoustic spatial localization and… · p. 10

F. Optical validation and performance metrics

  • Acoustic directions are rotated into the camera frame and projected onto synchronized video; a subset intersecting an animal can assign its whole clustered train.Frame-by-frame inspection finalizes assignments and links sequences to observed physical behaviors.Source: F. Optical validation and performance… · p. 10
  • DER_acoustic counts wrong-emitter and unassigned validated clicks against all validated scene clicks, but ignores over-segmentation and detection errors.It is therefore a clustering-specific bound, not directly comparable to standard speech DER.Source: F. Optical validation and performance… · pp. 10–11
  • VCR reports projected-localization concordance within 10° of the junk separately by image axis; a GMM partitions 2D projection errors for interpretation using acoustic and kinematic metadata.The GMM component count is selected using BIC and stability across 50 random initializations.Source: F. Optical validation and performance… · p. 11

III. RESULTS

The analyzed scenes span a measurable complexity gradient, from a single-emitter sequence to sustained overlapping sparring. Manually validated clicks and a filtered baseline pool define the dataset for subsequent evaluation.

A. Dataset validation and overall acoustic complexity

  • Manual review of 3,161 automatic detections yielded 2,655 validated true clicks; detection recall was 96.16% and precision was 80.77%.The 608 false positives were primarily surface noise and were discarded during validation.Source: A. Dataset validation and overall… · p. 11
  • Across nine scenes (1,368 clicks), ICI CV ranged from 0.152 in single-emitter Scene 7 to 0.567–0.720 in six overlapping sparring scenes; CV ≥0.5 marked multi-emitter scenes.Scenes 1 and 2 were moderately complex, with CV ≈0.49.Source: A. Dataset validation and overall… · pp. 11–12, Table I · p. 12
  • Filtering produced a 1,242-click baseline pool across 2.25 minutes, including focal and non-focal emissions, after excluding 126 low-energy focal-scene clicks.The global energy threshold reduced the 2,655 validated clicks to 2,319 before the focal-scene mask was applied.Source: A. Dataset validation and overall… · p. 12

B. Acoustic clustering and deinterleaving performance

Acoustic clustering separated many overlapping click trains, but performance weakened during temporal superpositions and fragmented scenes. Spatial localization and synchronized-video validation supported individual attribution for retained tracks, while vertical ambiguity limited some assignments.

B. Acoustic clustering and deinterleaving performance

  • 25.79% scene-level DER_cluster in Scene 4 was the peak; Scenes 1, 2, and 7 had 0.0%, while Scenes 8 and 9 had 5.59% and 6.98%.The baseline used CQT spectral extraction and ICI rhythmic tracking; resolving specific overlaps for final separation also required energy differentials and track-junction consistency.Source: B. Acoustic clustering and… · p. 12
  • CQT similarity stayed high within tracks but lost discriminative power during perfect temporal superpositions; raw spectra also varied across near-surface contexts.Similarity scores ranged from 0.63 to 0.87; Scene 1 showed pronounced notching while Scene 7 had more continuous broadband energy, reflecting sensitivity to propagation and array conditions.Source: B. Acoustic clustering and… · pp. 13–14, Figure 2 · p. 14
  • ICI tracking was less stable with similar rhythms or variable decelerations, and Scene 3’s combined loss of temporal and energetic discrimination left clustering reliant on masked CQT features.Scene 4’s erratic ICI accompanied the peak error; in Scene 9, energy differences resolved temporal ambiguity, whereas Scene 3’s tracks had similar ICIs and received energies.Source: B. Acoustic clustering and… · p. 14
  • Scenes 5 and 6 were excluded from final assignment because fragmented acoustic tracks and spatial ambiguities prevented continuous reconstruction and definitive attribution.The IPI was also unreliable for this dataset: same-sized focal subjects had indistinguishable theoretical IPIs, while depth and surface interactions produced aberrant or unreadable pulse structures.Source: B. Acoustic clustering and… · p. 15, C. Spatial localization, optical… · p. 15

C. Spatial localization, optical validation, and final individual attribution

  • KDE distinguished spatial modes for interacting emitters, including in entangled scenes, and provided the acoustic angles used for optical validation.In Scene 3, Track 3 peaked at -15.1° azimuth and 78.9° elevation, versus -4.1° and 72.5° for Track 4; in Scene 8, the tracks were at 12.4° and -1.4° azimuth.Source: C. Spatial localization, optical… · pp. 15–16, Figure 3 · p. 16
  • 100% horizontal visual concordance was reached for several primary tracks, while vertical concordance fell to 21.90% for Track 5.The reported rates came from projecting 637 aligned click coordinates onto synchronized video; the table summarizes spatial projection, concordance, and attribution for filtered focal tracks.Source: C. Spatial localization, optical… · p. 17, Table III · p. 17
  • Four GMM regimes classified the 637 spatially filtered clicks, from nominal success to proportional, body-shadowing/multipath, and horizontal-failure errors.Type 1 contained 361 clicks; Type 2 included 145, with 68 of Track 5’s 105 clicks affected and vertical concordance at 21.90%.Source: C. Spatial localization, optical… · p. 18, Figure 4 · p. 19
  • Ali and Daren were assigned as primary, secondary, or sole emitters across the retained vocal-sparring scenes.Ali was primary or sole in Scenes 1, 2, 7, and 8, and dominant in Scene 3; Daren dominated Scenes 4 and 9.Source: C. Spatial localization, optical… · p. 21

A. Overcoming near-field separation and attribution limits

Near-surface conditions undermine conventional acoustic separation and localization. The analysis characterizes spatial error regimes and identifies geometric and signal-related constraints on individual attribution.

A. Overcoming near-field separation and attribution limits

  • Near-surface spatial tracking is vulnerable to sensor-response mismatch, while pulse smearing and multipath make IPI separation unreliable for overlapping emitters.The array combines a dedicated reference channel for spectral and temporal extraction with heterogeneous sensors for spatial tracking.Source: A. Overcoming near-field separation… · p. 21
  • The algorithm couples ICI rhythmic predictions with CQT spectral consistency, using a first-order derivative as pre-whitening to expose abrupt click onsets.CQT loses discriminative power during perfect temporal superpositions, where the dominant emitter deforms the weaker track's patches.Source: A. Overcoming near-field separation… · pp. 21–22
  • Scenes 5 and 6 were excluded from primary performance tables because vertical alignment created overlapping spatial isopleths despite acoustic clustering assignments of 83.6% and 80.0%.Supplementary Table S3 retains the clustering metrics for these boundary cases.Source: A. Overcoming near-field separation… · p. 22

B. Physical and algorithmic origins of spatial errors (GMM interpretation)

  • 21.9% vertical VCR for specific tracks exposes a systematic elevation bias in spatial errors.The GMM interpretation links spatial errors primarily to TDOA degradation under complex acoustic conditions.Source: B. Physical and algorithmic origins of… · p. 22
  • Type 2 errors arise from proportional projection distortion at the field-of-view periphery, while Type 3 errors reflect directional bias from body shadowing and multipath.The Type 2 regime has n=145; Type 3 has n=98 and is associated predominantly with oblique postures.Source: B. Physical and algorithmic origins of… · p. 22
  • Type 4 correlation phantoms are systematic horizontal displacements during click collisions, a generalized cross-correlation limit without sufficient temporal separation.These errors affect specific entangled segments; overall spatial identity can remain resolvable.Source: B. Physical and algorithmic origins of… · p. 23
  • Near-planar geometry, signal saturation, and body shadowing create physical ambiguities that acoustic metrics alone cannot resolve, motivating optical validation.Vertical subject alignment can induce elevation blinding.Source: B. Physical and algorithmic origins of… · p. 23

C. Physical origin of the systematic elevation offset: array geometry and the 𝑷𝟎/𝑷𝟏

The acoustic elevation estimates sit 5°–8° above the visual reference, and two possible mechanisms could explain the offset. The dataset cannot distinguish them, so greater vertical array aperture is needed to resolve the ambiguity.

Physical origin of the systematic elevation offset

  • A +5° to +8° vertical offset persists against the distal-junk visual reference, making a 6° tolerance too narrow and motivating the 10° threshold.The 10° threshold is the minimum that encompasses the offset while spanning 8.5% of horizontal and 14.5% of vertical FOV.Source: hypothesis · p. 23
  • Two non-exclusive explanations are near-planar array geometry and tracking the initial P₀ pulse rather than the P₁ beam.The authors present the P₀/P₁ account as a physically motivated hypothesis, not a demonstrated result.Source: hypothesis · pp. 23–24
  • Chronological locking, off-axis loss of P₁ energy, and P₀ wavefront integrity could each favor TDOA tracking of P₀.P₀ arrives first and can trigger adaptive detection; off-axis posture weakens P₁, while P₀'s less-dispersed wavefront supports sharper cross-correlation.Source: hypothesis · pp. 23–24
  • Spatial errors concentrate in subsurface session 081356, consistent with multipath effects and potentially clearer P₀/P₁ separation reinforcing first-arrival locking.The passage treats these patterns as consistent with both proposed explanations, not as a way to distinguish them.Source: hypothesis · p. 24
  • Distinguishing geometry from anatomy requires a volumetric array with greater vertical aperture or a controlled source at known depths.Both explanations predict a similar positive vertical shift.Source: hypothesis · p. 24

D. Spectro-temporal variability and behavioral implications

Vocal-sparring click trains have variable spectral structure that can reflect propagation and animal movement as well as vocal behavior. The authors therefore pair acoustic analysis with optical kinematics and urge caution in interpreting off-body spectral modulation.

Behavioral implication

  • Attributing overlapping click trains to the interacting whales supports their association with vocal sparring rather than background activity, but the evidence covers one dyad and context.The authors say generalization requires larger, multi-individual datasets.Source: D. Spectro-temporal variability and… · p. 24

Spectro-temporal variability

  • CQT analysis resolves spectral structure that standard STFTs blur, while lower fluctuations below 5 kHz are likely mechanical rather than biological.Multipath, body masking, and changing off-axis orientation can alter the received spectrum of these broadband, variably rhythmic signals.Source: D. Spectro-temporal variability and… · p. 24

Depth-dependent propagation

  • Near the surface, interactions within 1 m show substantial attenuation and compressed reflection delays, whereas at 15 m delayed reflections restore low-frequency energy.The authors propose blind deconvolution for future behavioral signal validation to remove the environmental multipath response.Source: D. Spectro-temporal variability and… · p. 25

Behavioral interpretation

  • In array recordings, apparent spectral modulations can arise from propagation or pass-by kinematics rather than active vocal control.The authors caution against interpreting off-body array recordings as evidence of voluntary vocal modulation.Source: D. Spectro-temporal variability and… · p. 25
  • Optical orientation and jaw-kinematic ground truth can help separate off-axis propagation effects from genuine vocal modulation.The authors describe this as a technical proof of concept and call for larger joint kinematic-acoustic datasets.Source: D. Spectro-temporal variability and… · p. 25

V. CONCLUSION

The workflow assigned 11 intertwined click trains, totaling 882 clicks, to two immature males despite spatial ambiguities in compact-array estimates. The conclusion frames this as a technical proof of concept and the supplementary material documents its hardware, calibration, detection, and tracking procedures.

V. CONCLUSION

  • The multimodal workflow assigned 11 intertwined click trains totaling 882 clicks to immature males Ali and Daren, using visual concordance to resolve spatial ambiguities.The authors note that acoustic TDOA estimates alone were affected by mechanical biases.Source: §V · p. 25
  • The isolated signals add a non-coda acoustic behavior to the account of sperm-whale social exchanges, and larger datasets are needed to assess context-dependent modulation.The authors connect this work to future study of socio-acoustic networks and vocal ontogeny.Source: §V · p. 25
  • Attribution succeeded despite distorted raw waveforms; structural signal characterization still requires reconstruction to separate biological modulation from physical artifacts.The authors propose combining optical kinematics with anatomical acoustic models as one route toward informed deconvolution.Source: §V · p. 26

SUPPLEMENTARY MATERIAL

  • Supplementary materials document array hardware, calibration, threshold optimization, tracking parameters, validation and ICI statistics, excluded-scene metrics, and additional scene workflows.The cited supplementary figures cover scenes 1, 2, 4, 7, and 9.Source: SUPPLEMENTARY MATERIAL · p. 26

S1. OPALE array hardware specifications and geometry

  • The 2023 deployment used three heterogeneous hydrophones from a five-hydrophone OPALE array, with two defining the horizontal axis and one positioned lower and forward.The SQ26 sensors cover 20 Hz–45 kHz; the C75 sensor covers 10 Hz–170 kHz, and acquisition used 256 kHz sampling at 24-bit resolution.Source: S1. OPALE array hardware… · p. 26
  • The array coordinate matrix specifies the three sensor positions used by the TDOA localization algorithms.The local axes are forward, lateral-left, and upward.Source: S1. OPALE array hardware… · p. 27

S2. Acoustic calibration and detection parameters

  • C75 calibration applied a linear amplitude factor of 3 and polarity inversion before TDOA calculation, alongside sensor-specific sensitivity corrections.The passage reports sensitivities of −169 dB re 1V/µPa for SQ26 and −180 dB re 1V/µPa for C75.Source: S2. Acoustic calibration and detection… · p. 27

S3. ROC curve analysis and threshold optimization

  • Sensor-specific two-stage thresholds used permissive detection for track continuity and stricter thresholds for localization, while automatic detection reached 96.16% recall at the expense of precision.The stricter thresholds aimed for near-zero constant false alarm rate; manual validation removed false positives, primarily surface noise.Source: S3. ROC curve analysis and threshold… · p. 27, Table S1 · p. 28

AUTHOR DECLARATIONS

The authors report no conflicts and describe non-invasive observations conducted under ethical guidance and a research permit. Recordings, ground-truth data, code, and rendered assignment videos are available, but raw high-resolution video is not deposited.

AUTHOR DECLARATIONS

  • The authors disclose no conflicts of interest.Source: AUTHOR DECLARATIONS · p. 35
  • The study used non-invasive underwater observations without capture, tagging, or manipulation, and followed the stated ethical principles and Mauritian research regulations under permit.Source: AUTHOR DECLARATIONS · p. 35

DATA AVAILABILITY

  • Raw acoustic recordings, ground-truth datasets, analysis notebooks, and rendered assignment videos are available in the named repositories, but raw high-resolution videos are not deposited.Source: DATA AVAILABILITY · pp. 35–36

APPENDIX

Table IV traces click processing to the final focal subsets, while Figure 5 shows multimodal source separation in a highly entangled interaction where optical validation resolves acoustic ambiguity.

APPENDIX

  • 637 focal clicks form the final subset, intersecting 882 clustered clicks from resolvable sessions with 1,672 geometrically validated clicks.Session 085315, covering Scenes 5 and 6 and 240 clicks, was excluded from individual attribution because of unresolvable spatial ambiguities.Source: Table IV · p. 36
  • Table IV summarizes processing from raw detection to focal subsets, using distinct detection thresholds for clustering and spatial localization.The stages were conducted in parallel across three recording sessions.Source: Table IV · p. 36
  • Figure 5 shows acoustic waveform, energy, CQT, and spatial-distribution panels alongside video-plane validation for Ali and Daren.Dashed circles mark a 10° spatial tolerance around the distal end of the junk.Source: Figure 5 · p. 37
Loading 2609.12593v1…