PaperlayerView exact revision on arXiv

Source-linked AI summary

SEA-LM: Egocentric Spatial Audio Understanding for Wearable Microphone Arrays

Sonal Kumar, Sinan Hersek, Artem Dementyev, Mengzhen Pan, Ishan Chatterjee, Anurag Kumar, Ramani Duraiswami, Dinesh Manocha, Andrea Colaco

arXiv:2610.05610v2cs.SDcs.AI

TL;DR

Mono-channel audio-language models lack spatial awareness, limiting their ability to localize sounds and disentangle overlapping sources. SEA-LM combines a layout-flexible spatial encoder with semantic audio features and trains across six tasks; it improves most reported metrics over baselines and performs robustly across evaluated glasses arrays.

  • Problem

    Mono-channel audio-language models discard spatial cues needed for sound localization and overlapping-source disentanglement, while existing spatial MLLMs have limitations in hardware generalization, realism, and ego-centric modeling.

    Source: §1 · p. 1

  • Method

    SEA-LM uses FOACODER to encode FOA from variable smart-glasses arrays and combines its spatial embeddings with semantic audio features in an MLLM trained across six tasks.

    Source: §1 · p. 2

  • Results

    SEA-LM outperforms monaural and SELDNet-based baselines on most reported metrics; on Task 1, azimuth MAE drops from 80.6° to 14.9° versus the SELDNet-based baseline.

    Source: §1 · p. 2, §7 · pp. 9–10

  • Takeaways & Limitations

    SEA-LM maintains reliable wearer-speech understanding under dense source overlap and operates without array-specific retraining across 1,211 evaluated glasses arrays with 4 to 9 microphones.

    Source: §7 · p. 10

    Evaluation focuses on simulated, stationary scenes and 1,211 configurations from two glasses geometries; real and dynamic scenes, held-out topologies, and other device types remain for future work.

    Source: §7 · p. 10

Abstract

from arXiv · show

Embodied, ego-centric intelligence fundamentally requires the ability to comprehend spatial audio within complex environments. While large audio-language models excel at mono-channel reasoning, they lack spatial awareness, discarding critical spatial cues that enable sound localization and that can improve the disentanglement of overlapping sound sources. To address this, we present SEA-LM, a Spatial Audio Understanding model. First, we introduce FOACODER, a layout-flexible spatial audio encoder trained on source localization and ego-centric voice activity detection objectives to encode First Order Ambisonics derived from variable-count, variable-position smart-glasses arrays via beamforming. We train a Multimodal Large Language Model (MLLM) to understand these spatial audio embeddings through a two-stage curriculum spanning six tasks, including sound localization and spatially selective transcription in settings with multiple speakers and overlapping sounds. To prevent the transcription outputs from dominating the next token prediction loss and overwhelming the direction predictions, we introduce a Spatio-temporal Weighted Cross-Entropy Loss. On our evaluation set, SEA-LM achieves lower azimuth and elevation MAE, higher temporal IoU, lower external-source hallucination and missing-source rates, and lower WER on most transcription tasks than compared baselines, while remaining robust across 1,211 smart-glasses array configurations with 4 to 9 microphones.

1 INTRODUCTION

SEA-LM addresses the loss of spatial information in mono-channel audio-language models and gaps in existing spatial MLLMs. It combines a layout-flexible encoder, an ego-centric MLLM, and weighted training to support spatial audio understanding.

1 INTRODUCTION

  • Mono-channel audio-language models discard spatial cues that support speech intelligibility, overlapping-speaker disentanglement, and spatial awareness for embodied platforms.Source: §1 · p. 1
  • Existing spatial MLLMs face hardware and data-realism limits, often lack joint wearer-speech modeling, and can let dense transcripts overwhelm sparse direction and timestamp predictions.Source: §1 · pp. 1–2
  • SEA-LM combines FOACODER embeddings with semantic audio features, trains across six tasks, and uses weighted cross-entropy to prioritize sparse direction and timestamp tokens.The pipeline incorporates realistic array physics and FOA conversion artifacts; the model distinguishes wearer from bystander speech.Source: §1 · p. 2
  • Task 1 azimuth MAE falls from 80.6° to 14.9° and elevation MAE from 27.4° to 12.8° versus the SELDNet-based baseline across 1,211 glasses-array configurations.Location-bin accuracy rises from 42.0% to 84.1%.Source: §1 · p. 2

2 RELATED WORK

Prior audio-language models commonly rely on mono-channel inputs, while spatial-audio MLLMs address only parts of ego-centric scene understanding. SEA-LM is presented as jointly covering localization, temporal grounding, speech roles, and transcription.

Large Audio-Language Models

  • Mono-channel audio-language models discard ITD and ILD cues, limiting source localization and distance estimation and degrading overlapping-speaker disentanglement.Source: §2 · p. 3

Spatial Audio MLLMs

  • Existing spatial MLLMs vary in scope: some use binaural or qualitative FOA settings, while others omit wearable-array artifacts, speaker roles, or overlapping non-speech scenes.Source: §2 · p. 3
  • SEA-LM jointly targets continuous 3D localization, temporal grounding, speech and non-speech understanding, wearer–bystander attribution, and holistic and targeted transcription across realistic glasses arrays.Source: §2 · p. 3

3 METHODOLOGY

The methodology converts variable smart-glasses array audio into spatial embeddings and combines them with semantic audio features in SEA-LM. It trains the encoder and language model on simulated, overlapping scenes and six structured tasks.

3.1.1 PRE-PROCESSING; 3.1.2 SPATIAL AUDIO ENCODER ARCHITECTURE

  • ASM converts variable-count array audio to FOA; a raw microphone channel is added to counter beamforming information loss, and causal FOACODER encodes the spectrogram with 100-ms patches.FOACODER uses a causal self-attention backbone and produces 300 tokens of 512 dimensions for 30-second audio.Source: §3.1.1 · p. 4, §3.1.2 · p. 4

3.1.3 ENCODER TRAINING OBJECTIVES

  • FOACODER is jointly trained on frame-level wearer VAD and SELD objectives; its task-specific heads are then discarded, leaving a frozen encoder for SEA-LM.The SELD head uses two classes and three tracks per class; the combined VAD and SELD losses have equal weights.Source: §3.1.3 · p. 4

3.2.1 ARRAY TRANSFER FUNCTION SIMULATIONS; 3.2.2 ARRAY SELECTION

  • Array transfer simulations model propagation, scattering, and head occlusion; 1,211 sufficiently reconstructive four- to nine-microphone configurations are selected from two glasses geometries.Source: §3.2.1 · p. 5, §3.2.2 · p. 5

3.2.3 ROOM IMPULSE RESPONSE (RIR) GENERATION; 3.2.4 SOURCE AUDIO SAMPLING AND MIXING

  • Room simulations incorporate head and device scattering, while randomized wearer speech, bystander speech, and environmental sounds form dense overlapping scenes with spatial metadata.The pipeline generates approximately 200k/20k/20k RIRs and 2M/150k/150k 30-second examples for training, validation, and testing, respectively.Source: §3.2.3 · p. 5, §3.2.4 · p. 5

3.3.1 DUAL AUDIO PATHWAY

  • SEA-LM combines 750 native-audio embeddings at 40-ms resolution with 300 FOACODER embeddings at 100-ms resolution and text instructions in the language model.The native pathway supplies acoustic and semantic evidence; the spatial pathway supplies multichannel cues learned through SELD and wearer-VAD training.Source: §3.3.1 · pp. 5–6, §3.3 · p. 5

3.3.2 LEARNED SPATIAL PROJECTOR AND LLM INJECTION

  • A learned spatial projector maps FOACODER features into the LLM embedding space; invalid frames use a shared learned padding embedding while retaining fixed-length placeholders.Source: §3.3.2 · p. 6

3.3.3 SPATIAL AUDIO SUPERVISED FINE-TUNING

  • Supervised fine-tuning covers six tasks, from holistic localization and captioning to role-aware and targeted wearer or external transcription; selective tasks include absent-target examples.Source: §3.3.3 · p. 6

3.3.4 SPATIO-TEMPORAL WEIGHTED CROSS-ENTROPY

  • The weighted cross-entropy objective assigns larger weights to tokens aligned with spatial coordinates and event boundaries, countering transcript-heavy training targets.Prompt, multimodal-placeholder, and padding positions are excluded; spatial and temporal weights are each set to 5.Source: §3.3.4 · pp. 6–7

4 EVALUATION

The evaluation tests SEA-LM across six spatial-audio tasks on held-out simulated scenes, using task-specific metrics and controlled comparisons against three baselines.

4 EVALUATION

  • SEA-LM is evaluated end-to-end on six spatial-audio tasks, with FOACODER also assessed independently on SELD and wearer-VAD objectives.The independent encoder protocol and complete results are reported in Appendix D.Source: §4 · p. 7

4.1 SEA-LM EVALUATION SET

  • The held-out synthetic set contains 136,378 scene–question pairs across six tasks, using monophonic test audio and disjoint simulated RIRs.Tasks 2 and 6 include absent classes or directions in 30% of examples to test negative-query rejection.Source: §4.1 · p. 7

4.2 SEA-LM EVALUATION METRICS

  • Metrics span format adherence, source hallucinations and omissions, spatial and semantic accuracy, temporal localization, speech WER, and applicable wearer- or query-detection measures.External predictions are matched to ground truth by spatial and temporal agreement; wearer speech is matched directly because it has no spatial coordinates.Source: §4.2 · p. 7, Table 1 · p. 8

4.3 BASELINES

  • The three baselines are vanilla GEMMA-4-E4B-IT, fine-tuned monaural GEMMA, and SELDNet plus GEMMA, with the latter changing only the spatial encoder against SEA-LM.The controlled comparison retains the same LLM backbone, native audio encoder, synthetic SFT data, and two-stage curriculum.Source: §4.3 · p. 8

5 EXPERIMENTS

FOACODER receives joint VAD and SELD supervision, while SEA-LM uses a two-stage synthetic-data curriculum and controlled baselines that isolate spatial conditioning.

FOACODER training

  • FOACODER is trained for 2,000,000 steps with batch size 128 under joint wearer-VAD and SELD supervision.Training uses eight NVIDIA H100 GPUs.Source: §5 · p. 8

SEA-LM training

  • SEA-LM first trains its spatial projector on single-source Task 1 examples, then trains across all six tasks with one to four sources.Stage 2 updates the projector and rank-128 LLM LoRA adapters while freezing FOACODER, the native audio tower, and base LLM.Source: §5 · p. 8

Baseline training

  • The SELDNet baseline follows SEA-LM’s curriculum and training data but keeps its spatial encoder frozen, while the monaural baseline retains only the native audio pathway.These controls isolate the contribution of spatial audio conditioning.Source: §5 · p. 8

Results reporting

  • Table 2 presents task-wise evaluation results and identifies best and second-best values; its caption defines the reported spatial, temporal, source-type, and detection metrics.The caption names azimuth, elevation, distance, source-type accuracy, temporal IoU, hallucination, missing-source, and query F1 measures.Source: Table 2 · p. 9

6 RESULTS

SEA-LM performs best on nearly every reported end-to-end evaluation metric, while results also show strong encoder-level localization and wearer-VAD performance and identify limits under greater scene complexity.

6 RESULTS

  • The main-paper analysis centers on end-to-end SEA-LM results; independent FOACODER results and discussion are reported in Appendix D.Source: §6 · p. 9
  • 44.01° to 15.06° angular error and 32.29% to 86.50% localization recall are achieved by the selected 30-second FOACODER versus the encoder-only SELDNet reference, alongside 98.53% wearer-VAD F1.The reported encoder result indicates preservation of external-source geometry and egocentric speech activity before language-model integration.Source: §6 · p. 9

6.1 SEA-LM RESULTS

  • SEA-LM performs best on nearly every reported metric, reducing localization error, improving temporal boundaries, lowering external-source hallucinations and omissions, and improving wearer and external speech transcription.SELDNet + GEMMA is marginally better on Role-Aware Transcription WER.Source: §6.1 · p. 9
  • Performance degrades as source count and overlap increase, with the largest effect on external-speech transcription, while wearer-speech transcription remains comparatively robust.Across evaluated 4–9-microphone subsets, performance is stable, but fixed glasses geometries limit conclusions about arbitrary arrays or the general effect of adding microphones.Source: §6.1 · p. 9

7 CONCLUSION, LIMITATIONS & FUTURE WORK

SEA-LM combines spatial and semantic audio representations for ego-centric reasoning and outperforms monaural and SELDNet-based baselines on most reported metrics, with evaluation bounded to simulated stationary scenes and two glasses geometries.

Conclusion

  • The framework combines non-ideal FOA synthesis, FOACODER spatial embeddings, mono semantic audio features, and spatio-temporal weighted cross-entropy for sparse location and time supervision.The model is trained through a curriculum-learning approach.Source: §7 · p. 9
  • SEA-LM outperforms monaural and SELDNet-based baselines on most reported metrics and remains robust without array-specific retraining across 1,211 glasses arrays with 4–9 microphones.It maintains reliable wearer-speech understanding under dense source overlap.Source: §7 · pp. 9–10

Limitations and future work

  • The study evaluates simulated stationary scenes and 1,211 configurations from two smart-glasses geometries; real and dynamic scenes, held-out topologies, and other form factors remain future work.The authors also identify exhaustive component, uncertainty, and streaming studies as beyond scope.Source: §7 · p. 10

9 ETHICS STATEMENT

The study uses synthetic acoustic scenes and established datasets, not new human-subject data. It notes privacy risks and warns that corpus and simulation biases may produce uneven performance, requiring representative evaluation before consequential use.

9 ETHICS STATEMENT

  • Synthetic scenes and established datasets are used without new human-subject data collection.Source: §9 · p. 10
  • Spatial localization, role attribution, and transcription can support accessibility and embodied interaction, but may enable privacy-invasive monitoring or inference.Source: §9 · p. 10
  • Corpus, language, acoustic-environment, and simulated-geometry biases may yield uneven performance, so representative evaluation is needed before consequential use.The authors call for appropriate consent, data minimization, secure audio handling, and human oversight, especially in private or safety-critical settings.Source: §9 · p. 10

10 REPRODUCIBILITY STATEMENT

The paper maps its architectures, objectives, data generation, training, and evaluation procedures to main-text sections and appendices. It also specifies the coordinate convention, beamforming pipeline, array-selection constraints, FOACODER objectives, and training setup.

10 REPRODUCIBILITY STATEMENT

  • Appendices document the independent encoder evaluation, training stages, array and room simulation, source mixing, beamforming, task generation, evaluation protocol, cohort and curriculum analyses, extended results, and AI usage.Source: §10 · p. 10, APPENDIX CONTENTS · p. 14

A COORDINATE SYSTEM

B AMBISONIC SIGNAL MATCHING BEAMFORMING DETAILS

  • ASM computes beamforming weights from simulated transfer functions and spherical-harmonic targets, applies them to microphone signals, then inverse-STFTs the estimated FOA signal.The weights use Tikhonov regularization with λ = 10^-3 and are computed per array configuration.Source: B AMBISONIC SIGNAL MATCHING… · p. 15

D.1 TRAINING OBJECTIVES

  • FOACODER uses parallel VAD and SELD heads: VAD predicts wearer-speech presence, while SELD regresses 3D Cartesian directions for overlapping speech and non-speech tracks.The SELD setup uses two classes and three tracks per class, with Location-based Training addressing track permutation.Source: §D.1 · pp. 16–17

C BEAMFORMING ARTIFACTS AND ARRAY SELECTION

  • Coplanar arrays can attenuate or fail to reconstruct the elevation channel, while ASM can low-pass filter channels; selection therefore retains arrays meeting reconstruction thresholds.The 4-microphone coplanar configuration fails the stated NMSE criteria, whereas adding a fifth microphone with Z-axis displacement passes them.Source: C BEAMFORMING ARTIFACTS AND ARRAY… · pp. 15–16, Figure 3 · p. 17

D.1 TRAINING OBJECTIVES

  • The encoder’s joint pretraining loss weights VAD and SELD equally; padded frames are excluded from the localization loss, and task-specific heads are discarded after pretraining.Source: §D.1 · pp. 17–18

D.2 TRAINING SETUP FOR FOACODER.

  • FOACODER is trained for 2,000,000 steps with batch size 128 on eight NVIDIA H100 GPUs, using AdamW and a warmup-cosine learning-rate schedule.Source: §D.2 · p. 18

D.3 EVALUATION PROTOCOL

This section describes FOACODER’s encoder evaluation and supporting simulation and data-generation procedures, alongside role- and direction-targeted transcription outputs and the shared evaluation protocol and metrics for six tasks. It reports FOACODER’s results against SELDNet on localization and wearer-VAD metrics, noting that this is not a direct architectural benchmark.

D.3 EVALUATION PROTOCOL

  • Encoder evaluation uses 156,000 synthetic spatial-audio examples and reports DCASE SELD metrics, matching predicted and reference sources frame-by-frame with the Hungarian algorithm.The encoder predicts speech and non-speech; reported SELD metrics include Angular Error, Error Rate, F-score, Localization Recall, and aggregated SELD Score.Source: §D.3 · p. 18
  • Wearer VAD is evaluated with per-example Accuracy, Precision, Recall, F1-score, and AUC, excluding padded audio from the 30-second clips.The VAD metrics are averaged across examples; threshold-based metrics use timeframes where unactivated logits cross the activation threshold.Source: §D.3 · p. 18

D.4 RESULTS

  • FOACODER performs significantly better than the SELDNet baseline across DCASE and VAD metrics, but the comparison is diagnostic rather than a direct architectural benchmark.The evaluation results are presented in Table 3; the authors note differences in training data, model capacity, and dedicated wearer-VAD training as possible contributors to the gap.Source: Table 3 · p. 19, §D.4 · p. 19
  • The 30-second FOACODER encoder performs moderately better and was selected for SEA-LM, while a 1-second version offers lower inference compute and latency.The shorter-context model has a similar parameter footprint, and its shorter sequences reduce Transformer computation; it was not used in downstream experiments.Source: §D.3 · p. 18, §D.4 · p. 19

F ARRAY TRANSFER FUNCTION SIMULATION DETAILS

  • Transfer-function simulations use acoustic reciprocity and BEM to derive far-field responses, while mouth responses are spatially averaged over the mouth surface.Far-field responses are sampled on a 12,602-point sphere over 0–359° azimuth and −90–90° elevation; post-processing normalizes responses, converts them to impulse responses, and applies a Hann fade-out.Source: §F.1 · pp. 20–21, §F.2 · p. 21

G AMBISONIC SIGNAL MATCHING SIMULATION STUDY

  • The Ambisonic Signal Matching study uses 240 uniformly distributed far-field sources and NMSE to assess beamforming artifacts and guide microphone-array selection.RMS power spectra are also examined for spectral coloration; microphone noise is omitted from the simulated signal model.Source: G AMBISONIC SIGNAL MATCHING SIMULATION… · pp. 21–22

H.1 SIMULATING EXTERNAL SOURCES

  • The self-speech room response combines an accurate mouth-to-microphone path with reflections isolated by subtracting an anechoic response from a room simulation.External-source transfer functions use the nearest direction in the precomputed spatial grid when the exact direction is unavailable.Source: §H.1 · p. 22, §H.2 · p. 23

J QUESTION AND ANSWER GENERATION DETAILS AND EXAMPLES

  • Synthetic scenes randomize room, array, source, and mixing conditions, and valid question-answer tasks are sampled from scene contents.Each scene contains up to three sources, including wearer speech, external speech, or other sounds; tasks cover holistic or selective localization and holistic captioning, with absent-source queries allowed.Source: §H.3 · p. 23, I MIXING PARAMETERS · pp. 23–24, J QUESTION AND ANSWER GENERATION… · pp. 24–25
  • Role-aware transcription retains wearer and bystander speech, with spatial metadata only for bystanders; targeted tasks return wearer-only speech or bystander speech from a queried direction.The targeted external task uses categorical directions, such as “left.”Source: J QUESTION AND ANSWER GENERATION… · p. 26
  • For negative queries in Tasks 2 and 6, the output schema represents no matching source as an empty sources list, without a separate natural-language denial.The specified response is {"sources":[]}.Source: J QUESTION AND ANSWER GENERATION… · p. 26

K SEA-LM EVALUATION PROTOCOL AND METRICS

  • All models use the same parser, source-assignment procedure, and aggregation rules; task-specific results for the six-task metric suite appear in Appendix P.The appendix specifies the complete metric suite used for the six tasks.Source: K SEA-LM EVALUATION PROTOCOL AND METRICS · p. 26

K.1 COMMON EVALUATION PROTOCOL

The evaluation protocol matches predicted external sources to references using spatial and temporal agreement, then computes task-specific metrics on accepted matches. Metric definitions and evaluation scope vary by task, with wearer speech handled separately from external sources.

K.1 COMMON EVALUATION PROTOCOL

  • External sources are matched by Hungarian assignment using a cost that equally weights normalized great-circle direction distance and temporal IoU; assignments above 0.7 are rejected.The threshold was chosen by manual inspection of more than 20 training examples, fixed before evaluation, and not subjected to broader sensitivity analysis; unmatched predictions and references count as hallucinated and missing sources.Source: §K.1 · pp. 26–27
  • Localization, timing, source-type, and transcription errors are calculated only on accepted matches, while wearer speech is matched directly and excluded from external-source hallucination and missing-source rates.A missing or hallucinated wearer source is instead handled by wearer-detection metrics.Source: §K.1 · p. 27

K.2 METRIC DEFINITIONS

  • Spatial metrics include circular azimuth error, absolute elevation and distance errors, percentage distance error, and exact-match direction or distance-bin accuracy; source-type accuracy uses canonicalized labels.Distance bins are close below 1 m, medium from 1 to 3 m, and far above 3 m; errors and bin accuracies are pooled over matched external sources.Source: §K.2 · pp. 27–28
  • Temporal metrics are start- and end-time MAE and temporal IoU, while WER is capped at 100% and computed over standardized transcripts for matched speech.Temporal metrics are pooled over accepted pairs, with the evaluated source types determined by each task.Source: §K.2 · p. 28
  • Accuracy, precision, recall, and F1 are computed from dataset-wide aggregate counts, with query detection scored at source level and wearer detection at example level.For query detection, unmatched predictions and references are false positives and false negatives; empty predictions for absent targets contribute true negatives.Source: §K.2 · p. 28

K.3 METRICS BY TASK

  • Task 1 evaluates all external sources, Task 2 restricts localization to a queried sound class, and Task 3 evaluates wearer and external sources for captioning, localization, timing, and transcription.Task 2 also reports source-level query detection; Task 3 reports example-level wearer detection.Source: §K.3 · p. 29
  • Task 4 evaluates wearer and bystander speech, Task 5 only wearer speech, and Task 6 bystander speech in a queried direction, with metrics tailored to each scope.Task 6 includes source-level query detection, while Tasks 4 and 5 include example-level wearer detection.Source: §K.3 · p. 29

L COHORT ANALYSIS

The cohort analysis examines performance across scene complexity, source overlap, and array microphone count. Localization, temporal, transcription, and query-detection performance generally worsen as external-source complexity rises, while wearer transcription is less affected; tested microphone-count cohorts remain stable.

L COHORT ANALYSIS

  • Scenes are grouped by total active sources, speech speakers, microphone count, or the percentage of active time with two or more concurrent sources.Overlap is measured relative to total active duration and partitioned into five cohorts from zero to 100%.Source: L COHORT ANALYSIS · p. 29
  • As source count, speaker count, and source overlap increase, external-source localization, temporal grounding, transcription, and query detection generally worsen, while wearer transcription is comparatively stable.The three-or-more-speaker and highest-overlap cohorts contain fewer evaluated examples, making their estimates less stable than those for lower-complexity cohorts.Source: L COHORT ANALYSIS · pp. 30–31, Figure 5 · p. 30, Figure 6 · p. 31
  • Across the tested 4–9-microphone cohorts, localization, temporal IoU, transcription WER, and query detection remain stable, but this does not establish generalization to unseen array topologies.The evaluated positions come from subsets on two fixed glasses frames, limiting spatial spread relative to fully randomized or non-glasses layouts.Source: L COHORT ANALYSIS · p. 31

M CATEGORICAL DIRECTION BINS

N TRAINING STAGES FOR SEA-LM

SEA-LM uses a two-stage curriculum on synthetic training and validation partitions without external benchmark data. It first aligns the spatial projector on single-source localization, then jointly fine-tunes across six tasks and scenes with one to four sources.

N TRAINING STAGES FOR SEA-LM

  • Stage 1 trains only the spatial projector for three epochs on single-source Holistic Sound Localization, with the LLM, audio towers, FOACODER, token embeddings, and language-model head frozen.The projector is trained from a randomly initialized state while the other listed components remain fixed.Source: N TRAINING STAGES FOR SEA-LM · p. 33
  • Stage 2 continues projector training and adds LLM LoRA adapters, sampling all six tasks and scenes with one to four sources while keeping the base LLM and audio encoders frozen.The curriculum uses synthetic partitions and does not mix external benchmark data.Source: N TRAINING STAGES FOR SEA-LM · pp. 32–33

O.1 SEA-LM CURRICULUM ABLATION

The curriculum ablation reports that inserting a focused intermediate stage degrades nearly every reported metric, with the authors proposing several interacting explanations. The extended-results section points readers to complete task results, and the AI-usage statement describes writing and research-support uses.

O.1 SEA-LM CURRICULUM ABLATION

  • Adding a focused Stage 2a for selective and targeted tasks on one- or two-source scenes degrades nearly every reported metric.The authors hypothesize that the restricted task distribution may specialize the projector and LoRA adapters, that frozen audio representations may constrain alignment, and that a more complex optimization trajectory may hurt the outcome.Source: §O.1 · p. 34

P EXTENDED SEA-LM EVALUATION RESULTS

Q AI USAGE

  • The authors report using LLMs for grammar checks, word choice, sentence rewriting, literature discovery, and data curation.Source: Q AI USAGE · p. 34
Loading 2610.05610v2…