PaperlayerView exact revision on arXiv

Source-linked AI summary

Long-Term Memory-Guided Enhancement for Target Perception in Audio-Language Models

Zhenhong Zhou, Xuanyue Zhao, Youji Liu, Yuanhe Zhang, Xiaoyu Ma, Lianyu Hu, Yang Liu

arXiv:2609.36577v1cs.SDcs.AIcs.CL

TL;DR

ALLMs can lose the ability to perceive a target when noise and competing sources mix with it. LTM-AE uses clean-reference category memories to refine incoming audio representations without training the model. Across three ALLMs, classification gains averaged 29.53–46.15 percentage points over raw mixtures.

  • Problem

    ALLMs may struggle to perceive a specified target when stronger competing sources obscure it, particularly for non-speech sounds.

    Source: §1 · pp. 1–2

  • Method

    LTM-AE stores category structure from clean audio hidden states and uses the selected memory to reconstruct and interpolate incoming audio tokens without training the ALLM.

    Source: §1 · p. 3, §3.1 · p. 4, §3.2 · p. 4

  • Results

    Across three ALLMs, mean gains across two language-output protocols were 29.53–46.15 percentage points.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    The results show that long-term auditory experience can guide selective target perception in complex mixtures, including diverse non-speech sounds.

    Source: §5 · p. 9

Abstract

from arXiv · show

Audio large language models (ALLMs) can reason about the content of audio recordings to perform complex tasks. However, these capabilities usually collapse in real-world environments when background noise and competing sources mix the target sound. Inspired by long-term memory in human listening, we propose Long-Term Memory-Guided Audio Enhancement (LTM-AE) to improve selective target perception by refining the audio representations of ALLMs without training. LTM-AE extracts representations in hidden states from separate clean reference recordings as long-term memory for each category, guiding enhancement toward a user-specified listening target. We reconstruct incoming audio tokens in the selected category long-term memory and interpolate the reconstructions with the original tokens before language backbone decoding. This interpolation controls the influence of stored auditory experience while keeping all ALLM parameters fixed. Diagnostic readouts across twenty sound categories and three ALLMs show that LTM-AE strengthens responses to a specified target amid three interfering sources. Averaged over constrained and free-form classification, accuracy gains over raw mixtures range from 29.53 to 46.15 percentage points across multiple open source models. For speech content recovery, LTM-AE with an additional learned token-level gate reduces Qwen2-Audio's word error rate from 23.07% to 14.77%. This work takes an initial step toward using principles of human long-term memory to enhance ALLMs for real-world listening. Our code is available at https://github.com/aynlp/ltm-audio-code

1 INTRODUCTION

ALLMs can struggle to perceive a specified target when stronger sources interfere, motivating a way to use prior auditory experience for selective listening. LTM-AE uses category-specific memory to refine audio representations without training, and the paper reports gains across models and tasks.

1 INTRODUCTION

  • In real-world acoustic environments, competing sources can obscure targets, and ALLMs may struggle especially when the target is a non-speech sound.Human listening research motivates using prior auditory experience to guide perception toward a predefined target.Source: §1 · pp. 1–2
  • LTM-AE selects category-specific long-term memory from clean reference recordings to reconstruct and interpolate incoming audio tokens before decoding, with ALLM parameters fixed.The method calibrates retained rank and interpolation weight using paired clean and mixed validation recordings.Source: §1 · pp. 2–3
  • Mean language-output gains across two classification protocols range from 29.53 to 46.15 percentage points across the three evaluated ALLMs.The benchmark includes twenty target categories and mixtures with three interfering sources.Source: §1 · p. 2
  • LTM-AE with a learned token-level gate reduces Qwen2-Audio’s word error rate from 23.07% to 14.77%.The speech-transcription extension evaluates recovery of words not specified by the target category.Source: §1 · p. 3

2 RELATED WORKS

Related work traces memory research from human auditory experience to memory mechanisms in language models, alongside a progression from signal denoising to methods that address interference in ALLMs.

2.1 MEMORY IN HUMANS AND LANGUAGE MODELS

  • Human-memory research spans perceptual, declarative, habit, and auditory memory, while language-model work uses retrieval, representation reuse, and retained experience to guide agents and reasoning.Examples include perceptual memory, auditory memory, and retrieval-based generation (Lewis et al., 2020).Source: §2.1 · p. 3

2.2 DENOISING AND ROBUSTNESS IN AUDIO LANGUAGE MODELS

  • Audio denoising progressed from statistical and neural signal restoration to ALLM approaches involving embedding denoising, task-aware enhancement, and noisy-input response alignment.The surveyed work also studies acoustic variation, noisy-domain adaptation, audio tools, and realistic-scene robustness.Source: §2.2 · p. 3

3 LONG-TERM MEMORY-GUIDED AUDIO ENHANCEMENT

LTM-AE uses category-specific long-term memory from clean audio to refine representations of a user-specified target in mixtures, without training the ALLM. It builds and calibrates that memory offline, then interpolates reconstructed audio tokens with the current input before backbone decoding.

3.1 PROBLEM SETUP

  • LTM-AE selects memory for the user-specified category, reconstructs incoming audio tokens, and replaces them with interpolated refined tokens before backbone decoding.The audio-token sequence length and order remain unchanged, and the language backbone receives no separate target cue.Source: §3.1 · p. 4

3.2 RETAINING LONG-TERM AUDITORY EXPERIENCE

  • Category memory summarizes clean reference-token embeddings with their mean and dominant directions of variation, retaining a shared reference point and within-category variation.The mean and ordered directions are saved for reuse; retained ranks use prefixes of the same basis.Source: §3.2 · p. 4

3.3 REFINING AUDIO TOKENS WITH CATEGORY LONG-TERM MEMORY

  • Each incoming token is reconstructed using its coefficients along the retained category directions, then interpolated with the original token using a category-specific weight.The retained component is preserved while the orthogonal residual is scaled by 1−αc; the rank and weight govern what variation is preserved and how strongly the remainder is attenuated.Source: §3.3 · pp. 4–5

3.4 CALIBRATING LONG-TERM MEMORY GUIDANCE

  • Calibration selects the retained rank and interpolation weight by comparing enhanced mixture tokens with aligned clean-target tokens on a separate paired validation set.The selected settings are saved for reuse, while the category mean and directions remain fixed and ALLM parameters are not trained.Source: §3.4 · p. 5

3 LONG-TERM MEMORY-GUIDED AUDIO ENHANCEMENT

  • The full process builds category memory from clean embeddings, calibrates its settings, and applies the saved memory to the specified target before decoding, without training the ALLM.Inference uses the incoming audio tokens together with the saved memory and settings.Source: §3.4 · p. 5

4 EXPERIMENTS

Experiments test LTM-AE on twenty sound categories and three ALLMs, using constrained and free-form classification, retrieval, and speech transcription. LTM-AE improves target perception across models and readouts, while a learned token-level gate improves speech transcription on Qwen2-Audio.

4 EXPERIMENTS

  • The benchmark evaluates twenty sound categories spanning instruments, speech, animal vocalizations, and environmental events, with each mixture combining a target and three interferers.Aggregate accuracy comparisons use twenty separate clean reference clips per category to construct memory for each model, without training the ALLM.Source: §4 · p. 5

4.1 DIAGNOSTIC READOUTS OF SELECTIVE TARGET PERCEPTION

  • LTM-AE improves constrained classification across all three ALLMs; Qwen2-Audio accuracy rises from 9.60% on raw mixtures to 47.20% with LTM-AE, with unchanged prompts and no ALLM training.Free-form classification also improves across models: Qwen2-Audio rises from 4.80% to 26.25%.Source: §4.1 · p. 6
  • Retrieval accuracy for Qwen2-Audio rises from 8.25% on raw mixtures to 86.75%, approaching its 87.90% clean-recording accuracy.Retrieval bypasses language generation and measures alignment with clean recordings of the specified category before decoding.Source: §4.1 · p. 6

4.2 ANALYSIS OF ENHANCEMENT GAINS ACROSS READOUTS

  • 29.53 and 33.55 percentage points for Qwen2-Audio and Step-Audio-2-mini, respectively, and 46.15 for Kimi-Audio, are their mean language-output gains across constrained and free-form classification.LTM-AE improves all nine model-readout combinations; the smallest language-output gain is 21.45 points for Qwen2-Audio in free-form classification.Source: Table 2 · p. 7, §4.2 · p. 7
  • Retrieval gains exceed language-output gains for every ALLM, with Qwen2-Audio gaining 78.50 points in retrieval and Kimi-Audio gaining 46.15 points on average across language-output protocols.The gap varies by readout, but gains remain positive in both language-output protocols.Source: Table 2 · p. 7, §4.2 · p. 8
  • Category-level gains extend to non-speech sounds: retrieval is positive for every category across all three ALLMs, and car horn and glass breaking average gains of 78.0 and 77.3 points.For frog croak, retrieval gains reach 95, 99, and 97 points for the three ALLMs even though free-form classification does not improve for that category.Source: §4.2 · p. 8

4.3 EXTENDING LONG-TERM MEMORY GUIDANCE TO SPEECH TRANSCRIPTION

  • On the held-out test set, the selected token-level gate reduces Qwen2-Audio WER from 23.07% on raw mixtures to 14.77% across 880 reference words.Ungated LTM-AE yields 33.75% WER, exceeding the raw-mixture baseline.Source: §4.3 · p. 9, Table 3 · p. 9

5 CONCLUSIONS

LTM-AE uses category structure from clean audio hidden states as reusable guidance for refining incoming audio embeddings without training the ALLM. Across three ALLMs and twenty sound categories, it improves target perception in mixtures, with gains extending to speech transcription through a learned token-level gate.

5 CONCLUSIONS

  • Across three ALLMs and twenty sound categories, LTM-AE improves language outputs and aligns audio representations more closely with the specified target amid multiple-source interference.The gains include diverse non-speech sounds.Source: §5 · p. 9
  • LTM-AE retains category structure from clean audio hidden states and reuses it to refine incoming audio embeddings without training the ALLM.Source: §5 · p. 9
  • A learned token-level gate extends LTM-AE to speech transcription and improves recovery of words spoken in the current recording.Source: §5 · p. 9

AI USE STATEMENT

Generative AI tools assisted with literature retrieval, reference verification, manuscript consistency, methodology and reporting feedback, and drafting and revision. The authors reviewed the assistance and take full responsibility for the final work.

AI USE STATEMENT

  • Generative AI tools assisted with literature retrieval, reference verification, formatting and consistency review, methodology and reporting feedback, and manuscript drafting and revision.Source: AI USE STATEMENT · p. 9
  • The authors reviewed all AI-assisted outputs and suggestions.Source: AI USE STATEMENT · p. 9
  • The authors take full responsibility for the work’s final text, data, results, and claims.Source: AI USE STATEMENT · p. 9

A DETAILED TWENTY-CLASS RESULTS

The twenty-class appendix reports constrained, free-form, and retrieval results for Qwen2-Audio, Kimi-Audio, and Step-Audio-2-mini across memory reference counts. Across the models, language-output and retrieval readouts show gains for diverse sound categories.

A DETAILED TWENTY-CLASS RESULTS

  • The appendix presents class-wise results across twenty categories, three protocols, and evaluated clean-reference counts for the three ALLMs.The final row averages the twenty category accuracies.Source: A DETAILED TWENTY-CLASS RESULTS · p. 14

A.1 QWEN2-AUDIO

  • Qwen2-Audio improves mean accuracy across all three readouts at every evaluated reference count, with gains spanning musical and environmental targets.Glass-breaking constrained accuracy rises from 12% to 100%, while free-form accuracy rises from 4% to 50–85%; the corresponding tables report constrained, free-form, and retrieval results.Source: §A.1 · p. 14, Table 4 · p. 14, Table 5 · p. 15

A.2 KIMI-AUDIO

  • Kimi-Audio constrained accuracy rises from 8.30% on raw mixtures to 57.60–59.70%, while free-form accuracy rises from 4.90% to 43.30–45.80%.The class-wise results also include free-form gains for classical guitar, piano, and cat meows, and retrieval gains for cat meows.Source: §A.2 · p. 16, Table 7 · p. 16, Table 8 · p. 17

A.3 STEP-AUDIO-2-MINI

  • Step-Audio-2-mini mean constrained accuracy rises from 10.25% on raw mixtures to 42.05–45.80%, and free-form accuracy rises from 7.15% to 42.45–44.65%.Its retrieval accuracy rises from 10.45% overall to 78.50–83.65%; class-wise tables report results for all three protocols.Source: §A.3 · p. 18, Table 10 · p. 18, Table 11 · p. 19

B PROMPTS AND EVALUATION DETAILS

The appendix describes the prompts and scoring procedures for constrained and free-form classification, prompt-free retrieval, and the evaluation conditions used across models and protocols.

B PROMPTS AND EVALUATION DETAILS

  • The same prompts and candidate labels are used across all three ALLMs, but the selected enhancement category is not disclosed in the prompts.The listening cue selects the category memory and calibrated settings before enhanced embeddings are passed to the readout.Source: B PROMPTS AND EVALUATION DETAILS · p. 20

B.1 CONSTRAINED-CLASSIFICATION PROMPT

  • Constrained classification asks the model to choose exactly one label from a shared candidate set, and scoring checks whether it matches the specified target.Common answer codes make scoring independent of how the model would otherwise name the sound.Source: §B.1 · p. 20, Table 13 · p. 20

B.2 FREE-FORM CLASSIFICATION PROMPT

  • Free-form classification removes candidate labels and asks for a short sound description, which is mapped to a category using fixed keyword patterns.Responses matching zero or multiple categories do not count as correct.Source: §B.2 · p. 20

B.3 PROMPT-FREE RETRIEVAL

  • Prompt-free retrieval compares enhanced audio representations with a clean reference gallery to predict the target category without a text query or generated answer.It averages the five largest within-category cosine similarities; for LTM-AE, pooling follows enhancement and measures representations before language generation.Source: §B.3 · p. 21

B.4 EVALUATION CONDITIONS

  • Every model and evaluation protocol uses the same five conditions: clean target-audio classification or retrieval, unenhanced mixtures without target-category input, and PCA enhancement using 20, 50, or 100 clean memory examples per class.Source: §B.4 · p. 21
  • Accuracy is computed within each target category and averaged across twenty categories; each category contributes 100 test recordings.The macro-average equals accuracy over all 2,000 test recordings, and gains are differences from raw-mixture accuracy in percentage points.Source: §B.4 · p. 21

C ADDITIONAL METHOD DETAILS

LTM-AE operates on audio embeddings at each model’s audio–language interface, using clean-reference memory to guide token reconstruction. The appendix details the memory construction, calibration, token-level speech gate, and solver cross-check.

C.1 AUDIO INTERFACES AND MEMORY CONSTRUCTION

  • Each ALLM applies its own audio-interface mapping to clean references and mixtures, replacing audio embeddings while leaving text embeddings, sequence structure, and target cues unchanged.Qwen2-Audio uses post-projector encoder outputs, Kimi-Audio uses fused adaptor embeddings, and Step-Audio-2-mini uses adaptor outputs.Source: §C.1 · p. 22
  • Every valid reference token contributes equally to a category’s mean and dominant directions, which randomized PCA or full SVD estimates without temporal pooling or token normalization.The implementation retains ordered basis directions and uses their selected prefix for inference.Source: §C.1 · p. 22
  • Validation selects the retained basis rank and interpolation weight by comparing enhanced tokens with aligned clean targets; inference then uses the compact stored mean and basis.For T tokens, the additional computation is O(Tdkc), with O(d(kc + 1)) deployed storage independent of reference count N for fixed d and kc.Source: §C.1 · p. 22

C.2 EFFECT OF MEMORY GUIDANCE

  • The retained-direction component keeps weight one, while interpolation scales the orthogonal component by 1 − αc; calibration evaluates the full clean-target error when selecting rank and weight.At αc = 0 the token is unchanged; increasing αc moves it toward the memory reconstruction around the stored mean.Source: §C.2 · p. 23

C.3 TOKEN-LEVEL GATING FOR SPEECH TRANSCRIPTION

  • For speech transcription, a shared token-level gate uses each token and its memory correction to scale the correction, and training updates only the gate using reconstruction and transcription losses.Gate values of 0 and 1 respectively retain the original embedding or recover ungated LTM-AE; inference needs no clean target or reference transcript.Source: §C.3 · p. 23

C.4 PCA–SVD SOLVER CROSS-CHECK

  • PCA and full SVD selected the same rank and interpolation weight for all 60 Qwen2-Audio category–memory-size pairs and produced identical per-class retrieval accuracies.The cross-check covers 20 categories at N ∈ {20, 50, 100}.Source: §C.4 · p. 24, Table 14 · p. 24

D MULTIPERCEPTION EXPERIMENTAL RESULTS

  • MultiPerception combines multiple sound categories in each mixture to test specified-target enhancement amid competing sources, comparing reference counts and representative language outputs.The appendix describes analyses connecting aggregate readout gains with individual listening targets.Source: D MULTIPERCEPTION EXPERIMENTAL RESULTS · p. 25

D.1 DATA SOURCES AND MIXTURE CONSTRUCTION

MultiPerception combines source collections across speech, music, animals, and environmental sounds, with clean clips partitioned for memory construction, validation, and evaluation. Its mixtures pair each target with three interferers at a fixed overall interference level.

D.1 DATA SOURCES AND MIXTURE CONSTRUCTION

  • The twenty target categories combine source collections spanning musical instruments, human speech, animal vocalizations, and environmental events.Some categories combine several collections.Source: §D.1 · p. 25, Table 15 · p. 25
  • Each three-second, mono 16 kHz category has disjoint 100-clip memory, validation, and evaluation partitions, totaling 6,000 clean clips across categories.Audio checks and hash-based duplicate removal are applied; memory conditions use N ∈ {20, 50, 100} clips.Source: §D.1 · p. 25
  • Each target is mixed with three distinct non-target categories after RMS matching, then scaled to a −10 dB target-to-combined-interference ratio.This yields 2,000 validation and 2,000 evaluation mixtures, with the same evaluation audio used by all three ALLMs.Source: §D.1 · p. 25

D.2 RESULTS ACROSS MEMORY SIZES AND CATEGORIES

Across evaluated reference counts, LTM-AE gains persist in aggregate readouts, while representative cases show outputs shifting toward the specified sound. The piano-memory test reports no piano mentions in recordings without piano sounds.

D.2 RESULTS ACROSS MEMORY SIZES AND CATEGORIES

  • 86.10%–86.75% Qwen2-Audio retrieval versus 8.25% on raw mixtures, 43.30%–45.80% Kimi-Audio free-form accuracy versus 4.90%, and 78.50%–83.65% Step-Audio-2-mini retrieval versus 10.45% persist across all three reference counts.Figure 5 summarizes the highest macro accuracy among evaluated counts; it is not the fixed N = 20 comparison, and category-level maxima are selected separately.Source: §D.2 · p. 26, Figure 5 · p. 26

D.3 REPRESENTATIVE LANGUAGE-OUTPUT CASES

  • In held-out N = 20 cases, Qwen2-Audio changes from reporting competing dog sounds to the specified alarm, while Step-Audio-2-mini changes from violin to target flute.A constrained-output example also shifts toward the specified frog croak.Source: §D.3 · p. 27, Table 16 · p. 27

D.4 RESPONSE TO LONG-TERM MEMORY WHEN THE TARGET IS ABSENT

  • Zero of 100 recordings without piano sounds elicited a piano mention before or after applying piano long-term memory.The same free-form prompt was used in both conditions.Source: §D.4 · p. 27, Table 17 · p. 27

E SUPPLEMENTARY SPEECH TRANSCRIPTION RESULTS

  • The supplementary speech experiment extends category-level target responses to recovering words from the current recording, using a gate to adjust memory guidance at each audio token.The speech category specifies the listening target, while the transcript depends on the recording’s content.Source: E SUPPLEMENTARY SPEECH TRANSCRIPTION… · p. 28

E.1 VALIDATION SELECTION AND FINAL EVALUATION

The final gated Qwen2-Audio evaluation reports lower WER than both raw mixtures and ungated LTM-AE on matched test data. The gate’s loss weight was selected on development examples before held-out testing.

E.1 VALIDATION SELECTION AND FINAL EVALUATION

  • The Qwen2-Audio experiment uses 500 clean speech clips for memory construction, with k = 8 and α = 0.7, while keeping the category memory and ALLM parameters fixed.The speech gate is trained for four epochs; it receives each original token and its LTM-AE correction.Source: §E.1 · p. 28
  • The token-reconstruction loss weight is selected by a development sweep: 80 examples train each candidate and 20 select it, after which the chosen configuration is trained on all 100 development examples.The 100 held-out test mixtures are not used to select the loss weight.Source: §E.1 · p. 28
  • 14.77% WER with the selected gate compares with 23.07% raw-mixture WER and 33.75% ungated LTM-AE WER on the matched test set.The selected gate reduces WER by 8.30 percentage points relative to raw mixtures.Source: §E.1 · p. 28

Related papers

Loading 2609.36577v1…