PaperlayerView exact revision on arXiv

Source-linked AI summary

SAKE: Towards Editing Auditory Attribute Knowledge of Large Audio-Language Models

Chih-Kai Yang, Yen-Ting Piao, Tzu-Wen Hsu, Szu-Wei Fu, Zhehuai Chen, Ke-Han Lu, Sung-Feng Huang, Chao-Han Huck Yang, Yu-Chiang Frank Wang, Yun-Nung Chen, Hung-yi Lee

arXiv:2510.16917v2cs.SDcs.AIcs.CLeess.AS

TL;DR

Auditory perceptual knowledge editing remains underexplored, despite differing from updates to isolated facts because it must alter acoustic generalization. SAKE benchmarks this capability across four dimensions and multiple LALMs; results show reliable edits often fail to generalize or propagate, while modality-connector fine-tuning is a more balanced baseline.

  • Problem

    Auditory knowledge editing is largely unexplored, and it is unclear whether factual editing methods generalize to abstract auditory concepts that require modifying perceptual generalization.

    Source: §1 · p. 1

  • Method

    SAKE benchmarks auditory attribute editing in LALMs across reliability, generality, locality, and portability, evaluating eight editing methods on three models.

    Source: §1 · pp. 1–2

  • Results

    Most methods enforce edits but struggle with auditory generalization, intra-attribute locality, and connected-knowledge propagation; sequential editing often brings forgetting or degeneration.

    Source: §1 · p. 2, Qwen · p. 7, §6 · p. 9

  • Takeaways & Limitations

    Fine-tuning the modality connector is a strong, more balanced baseline for future auditory-specific editing work.

    Source: §1 · p. 2, §6 · p. 9

    Most evaluated methods struggle to preserve non-target auditory knowledge and propagate edits during reasoning, and often forget or degenerate under sequential editing.

    Source: §6 · p. 9

Abstract

from arXiv · show

Knowledge editing enables targeted updates without retraining, but prior work focuses on textual or visual facts, leaving abstract auditory perceptual knowledge underexplored. We introduce SAKE, the first benchmark for editing perceptual auditory attribute knowledge in large audio-language models (LALMs), which requires modifying acoustic generalization rather than isolated facts. We evaluate eight diverse editing methods on three LALMs across reliability, generality, locality, and portability, under single and sequential edits. Results show that most methods enforce edits reliably but struggle with auditory generalization, intra-attribute locality, and multimodal knowledge propagation, and often exhibit forgetting or degeneration in sequential editing. Additionally, fine-tuning the modality connector emerges as a more robust and balanced baseline compared with directly editing the LLM backbones. SAKE reveals key limitations of current methods and provides a foundation for developing auditory-specific LALM editing techniques.

1. Introduction

SAKE addresses the largely unexplored problem of editing abstract auditory perceptual knowledge, which requires changing how models generalize perceptual evidence rather than updating isolated facts. Its evaluation finds persistent challenges in generalization, preservation of non-target knowledge, and edit propagation, while modality-connector fine-tuning is a more balanced baseline.

1. Introduction

  • Auditory knowledge editing remains largely unexplored, and it is unclear whether factual editing methods generalize to abstract auditory perceptual concepts.Unlike isolated factual associations, perceptual editing changes high-level abstractions over signals with substantial intraclass variability.Source: §1 · p. 1
  • SAKE is introduced as the first benchmark for editing auditory attribute knowledge in LALMs, evaluating reliability, generality, locality, and portability.Its scope includes speaker gender, emotion, spoken language, and animal sounds.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • The paper positions auditory editing as relevant to correcting misrecognition, debiasing, and voice-assistant personalization.These applications are described as more challenging than text-based settings because of auditory modality complexity.Source: §1 · p. 2
  • Across three LALMs, existing methods struggle to preserve non-target auditory knowledge and propagate edits during reasoning, and most suffer severe forgetting under sequential editing.Fine-tuning the modality connector is a strong, more balanced baseline.Source: §1 · p. 2

2. Related Work

Prior knowledge-editing work targets factual knowledge and related applications, while recent LVLM benchmarks extend evaluation to visual facts. Auditory attribute editing in LALMs remains unexamined despite the modality's growing integration into language models.

2.1. Knowledge Editing

  • Knowledge-editing methods use hypernetworks, neuron updates, or in-context learning, and have been applied to factual correction, bias mitigation, detoxification, personalization, and unlearning.Visual benchmarks including MMEdit, VLKEB, and MC-MKE address visual factual editing; prior work had not examined auditory attribute editing.Source: §2.1 · p. 2

2.2. Large Audio-Language Models (LALMs)

  • LALMs typically combine auditory encoders with an LLM backbone, but little attention has been given to editing auditory-specific knowledge in these models.Prior visual editing benchmarks do not address abstract auditory concepts rather than concrete facts.Source: §2.2 · pp. 2–3, §2.1 · p. 2

3. SAKE: Speech & Audio Attribute Knowledge Editing Benchmark

SAKE formalizes auditory attribute editing as updating an LALM to produce a desired label for an audio-and-text input. It evaluates edit success, consistency across equivalent inputs, preservation outside the edit scope, and transfer to connected knowledge, using constructed datasets spanning auditory attributes and test conditions.

3.1. Problem Formulation

  • Given an auditory input, text input, and desired target, editing updates an LALM's parameters so it generates that target for the edited pair.The edited target is a new auditory attribute label, such as changing a speech sample's perceived emotion from happy to sad.Source: §3.1 · p. 3

3.2. Evaluation Dimensions and Corresponding Metrics

  • SAKE evaluates reliability on edit instances, generality across equivalent neighborhoods, locality on out-of-scope behavior, and portability across connected knowledge.Locality measures whether behavior remains unchanged outside the edit scope; portability tests whether updated knowledge generalizes to related knowledge.Source: §3.2 · p. 3

3.3. Dataset Construction

  • SAKE covers speaker gender, emotion, spoken language, and animal sounds, with 1,200 editing instances formed from 300 balanced label pairs per attribute.The benchmark uses speech and audio multiple-choice question answering, and training and test sets are disjoint for methods needing auxiliary data.Source: §3.3 · pp. 3–4
  • Generality tests equivalent neighborhoods through paraphrased text questions, alternative audio with the same label, and combined audio-and-text variations.The paraphrased data are manually verified.Source: §3.3 · p. 4
  • Audio locality tests whether edits spare other attributes, uninvolved labels within the same attribute, existing knowledge of the target label, and general auditory capability.Text locality uses MMLU question–answer pairs; portability questions exclude answers valid for both original and edited labels.Source: §3.3 · p. 4

4. Experimental Settings

The experimental settings compare eight editing approaches across three LALMs in single and sequential editing scenarios. A language-model judge evaluates the metrics, with human ratings used to validate its judgments.

4. Experimental Settings

  • Three LALMs—DeSTA2.5-Audio, Qwen2-Audio-Instruct, and Audio Flamingo 3—are evaluated under single and sequential editing with greedy decoding.Source: §4 · p. 4

4.1. Editing Methods

  • Eight methods span fine-tuning, hypernetwork, neuron optimization, in-context, and memory editing, including FT (LLM), FT (Audio), KE, MEND, UnKE, I-IKE, IE-IKE, and WISE.FT (LLM) updates the backbone’s last layer; FT (Audio) updates the modality connector.Source: §4.1 · pp. 4–5

4.2. Single Editing and Sequential Editing

  • Single editing tests one knowledge item, while sequential editing applies repeated edits across ten sequences of ten instances and tracks edit persistence by the gap between application and evaluation.The evaluation uses the first five edits and caps the gap at five to keep sample counts comparable across gaps.Source: §4.2 · p. 5, Figure 2 · p. 5

4.3. Evaluator

  • GPT-5 mini judges the metrics, with human evaluation of 420 randomly sampled cases reaching 98.10% agreement.Source: §4.3 · p. 5

5. Results

Across single edits, methods often enforce targets but struggle to generalize, preserve neighboring auditory knowledge, or propagate changes; sequential edits add forgetting and degeneration. FT (Audio) emerges as a comparatively balanced and robust baseline.

5.1. Single Editing

  • Parameter-updating methods mostly reach ≥95% reliability, often ≈100%, with WISE best; IKE variants underperform.Source: §5.1 · p. 5

Qwen

  • FT (Audio) preserves perfect text locality, while FT (LLM) degrades it on most LALMs; KE and MEND preserve it reasonably well through regularization.Source: Qwen · p. 6
  • Portability remains uneven: methods update some related-knowledge categories but leave others unchanged, rather than transferring edits uniformly.Source: Qwen · pp. 6–7

5.1. Single Editing

  • Most methods enforce edited behavior but struggle to generalize, propagate changes to connected knowledge, and preserve unrelated knowledge within the same auditory attribute.Source: Qwen · p. 7

5.2. Detailed Analyses of Different Types of Generality and Audio Locality

  • Type 3 generality, combining textual and auditory neighborhoods, is hardest; audio-locality type 2 and type 4 broader auditory processing are also difficult to preserve.Type 2 concerns unrelated knowledge within the same attribute; most methods show degradation in general auditory processing ability.Source: §5.2 · p. 7

5.3. Sequential Editing

  • Reliability and generality decline as edit gaps grow; MEND and KE often drop rapidly after a few edits, while MEND on DeSTA, KE, and UnKE on AF show degeneration.Source: §5.3 · pp. 7–8
  • IKE variants trade weaker single-edit performance for stability at larger gaps, while locality usually avoids sharp sequential drops and portability stays below the 27.5% random baseline for most methods.Source: §5.3 · p. 8
  • FT (Audio) remains relatively robust across most LALMs and metrics, with notable sequential reliability and generality degradation only on DeSTA.Source: §5.3 · p. 8

6. Conclusions

The paper establishes auditory attribute editing in LALMs as a research direction and introduces SAKE to evaluate it. Its experiments identify persistent editing challenges and position modality-connector fine-tuning as a balanced baseline.

  • SAKE is the first benchmark evaluating auditory attribute knowledge editing across four dimensions, and experiments reveal challenges in preserving non-target knowledge and propagating edits during reasoning.Source: §6 · pp. 8–9
  • Under sequential editing, most methods suffer severe forgetting or degeneration, while fine-tuning the modality connector serves as a strong, more balanced baseline.Source: §6 · p. 9
  • Many methods effective in textual domains degrade substantially on auditory attributes, highlighting distinctive challenges for auditory knowledge editing.Source: §6 · p. 9

Impact Statement

The impact statement frames auditory knowledge editing as potentially useful for correcting errors, mitigating bias, and personalizing LALMs, while recognizing misuse risks. It also describes the paper’s resources, data sources, and use of LLMs.

Impact Statement

  • The paper presents SAKE as a foundation for future research and says its related resources will be released after the review process.Source: Impact Statement · p. 9
  • Auditory knowledge editing may support error correction, bias mitigation, and personalization, but modifying perceptual knowledge also introduces risks of adversarial manipulation.Source: Impact Statement · p. 9

A. The Usage of Large Language Models (LLMs)

B.1. Audio Sources

  • Audios primarily come from SAKURA, compiled from Common-Voice, CREMA-D, ESC-50, and Animal-Sound Dataset; type 4 locality data also uses Dynamic-SUPERB Phase-2 samples.Source: §B.1 · p. 14

B.2. Dataset Statistics

The dataset statistics section points readers to training- and testing-set summaries, then describes the benchmark’s attribute coverage and model selection. The chosen attributes and three diverse LALMs provide a focused testbed for evaluating auditory editing.

B.2. Dataset Statistics

  • The training and testing dataset statistics are summarized in Tables 4 and 5, respectively.The tables are identified as dataset summaries; the supplied passages do not give their numerical contents.Source: §B.2 · p. 14, Table 4 · p. 14, Table 5 · p. 15

C.1. Attributes

  • SAKE covers animal sounds, speaker emotion, speaker gender, and spoken language as core, widely used audio-understanding attributes.The authors describe these attributes as supported by comparatively rich audio datasets and question–answer resources, including for multi-hop portability evaluation.Source: §C.1 · pp. 14–15

C.2. Models

  • The evaluation uses Qwen2-Audio, DeSTA2.5-Audio, and Audio Flamingo 3, spanning different model families and audio and language-model designs.The authors say key observations recur across these models and methods, supporting their use as a diverse testbed rather than a model comparison.Source: §C.2 · p. 15

D. Implementation Details of the Editing Methods

The section describes parameter-editing and in-context methods, their implementation choices, and the evaluation procedure. Human checks show high agreement with the LLM judge on sampled correctness and consistency decisions.

D.1. Editing Methods

  • The evaluated methods include gradient fine-tuning, hypernetwork editors, in-context IKE variants, and WISE’s dual-memory design.Fine-tuning targets either the LLM’s last layer or the modality connector; IKE is evaluated with instruction-only and instruction-plus-example variants.Source: §D.1 · pp. 16–17
  • UnKE first shifts a chosen layer’s hidden-state key vector toward the target, then updates that layer to produce the new vector naturally.The second stage uses preservation data to constrain drift; the supplied implementation passage specifies 20 text and 5 audio preservation samples.Source: §D.1 · p. 16, §D.2 · p. 18

D.2. Implementation Details

  • Implementation choices include method-specific stopping or training rules, preservation data, WISE routing parameters, and retrieved demonstrations for IE-IKE.IE-IKE demonstrations are retrieved by editing pair and cover reliability, generality, audio locality, and text locality queries.Source: §D.2 · pp. 17–18

E. More Details and Evaluation Prompts for LLM-as-a-judge

  • The LLM judge evaluates correctness and consistency, and human verification on 210 samples agreed at 98.57% and 97.62%, respectively.Exact matches to ground truth or original outputs are handled directly without invoking the judge.Source: E. More Details and Evaluation Prompts… · p. 18

F. Detailed Results for Each Auditory Attribute under Single Editing

Under single editing, reliability is generally high, but modality-connector fine-tuning has an emotion-generalization weakness. Portability favors that method across attributes, while locality varies by attribute and model; the next section presents sequential-editing tables and plots.

F. Detailed Results for Each Auditory Attribute under Single Editing

  • All methods except I-IKE and IE-IKE show consistently high reliability across attributes; the two IKE variants vary by attribute and model.The authors attribute the IKE variation to differences in models’ in-context learning ability.Source: F. Detailed Results for Each Auditory… · p. 20
  • FT (Audio) has an emotion-generality drop from 99.67% to 78.56% on DeSTA, 100.00% to 71.67% on Qwen, and 100.00% to 63.00% on AF.These are the largest such drops on DeSTA and Qwen, and the second-largest on AF, among attributes.Source: F. Detailed Results for Each Auditory… · p. 20
  • Audio locality varies by attribute and model, while text locality is comparable across attributes; gender averages are elevated because locality type 2 does not apply.On DeSTA, language has higher audio locality in types 2 and 3; on AF, emotion has lower type-2 audio locality across most methods.Source: F. Detailed Results for Each Auditory… · p. 20
  • FT (Audio) achieves higher portability scores across all attributes, while methods’ portability varies across attributes and LALMs.The authors suggest modality-connector editing may more effectively propagate edited knowledge to interconnected knowledge.Source: F. Detailed Results for Each Auditory… · p. 20

G. Detailed Results under Sequential Editing

  • Sequential-editing results are reported in Tables 11–13, with corresponding line charts in Figure 4a–c for DeSTA, Qwen, and AF.The supplied passage identifies the result displays but does not state their outcomes.Source: G. Detailed Results under Sequential… · p. 24

H. More Discussions of Intra-attribute Knowledge Entanglement

The discussion links difficult intra-attribute preservation to acoustic overlap and possible semantic coupling, while noting that current methods do not explicitly model fine-grained distinctions. Portability evaluation spans varied related concepts and shows inconsistent propagation, including category-level variation.

H. More Discussions of Intra-attribute Knowledge Entanglement

  • Audio Locality Type 2 is hardest to preserve, possibly because within-attribute distinctions are more tightly coupled than between-attribute distinctions.The authors hypothesize that acoustic and semantic factors may jointly contribute.Source: H. More Discussions of Intra-attribute… · p. 24
  • Labels within an attribute can share acoustic characteristics, such as sad and fearful speech sharing lower energy, tension, or slower rates, making neighboring labels vulnerable to unintended effects.The discussion also suggests fine-grained semantic cues may be closely distributed, with FT (LLM) performing substantially worse than FT (Audio) on Audio Locality Type 2.Source: H. More Discussions of Intra-attribute… · p. 24
  • Current methods do not explicitly model fine-grained intra-attribute structure; the authors identify better isolation and preservation of these distinctions as future work.They suggest acoustic overlap and semantic coupling may contribute to the observed preservation failures.Source: H. More Discussions of Intra-attribute… · p. 24

I. Detailed Breakdown of Portability Evaluation

  • Portability questions cover diverse related concepts for each attribute, and Figure 5 illustrates the knowledge-category distribution for animal sounds.The evaluation aggregates multiple instances sharing an edit pair while using different portability questions targeting varied related concepts.Source: §I · p. 28, §I.1 · p. 28, Figure 5 · p. 28

I.2. Performance Breakdown

  • The authors attribute portability failures to ineffective edit propagation rather than missing world knowledge, which their models already possess for the dataset.The connected attribute knowledge is drawn from SAKURA.Source: §I.2 · p. 29
  • 45.95% versus 7.89%: FT (Audio) portability accuracy differs between “reproduction” and “family” for Animal Sound, while failures to update related concepts remain the majority overall.Across categories, propagation is mixed rather than uniformly successful or unsuccessful.Source: §I.2 · p. 29, Table 14 · p. 30

J. Case Study

The case study contrasts a successful targeted emotion edit, which preserves locality and updates connected reasoning, with sequential editing that can produce incoherent outputs and broader failures. Quantitative degeneration patterns vary across methods and models.

Case Study

  • The case study contrasts a single targeted emotion edit with sequential edits across multiple concepts, highlighting reliability against accumulating errors.Source: J. Case Study · p. 34
  • FT (Audio) on Qwen changes a speaker-emotion prediction from “fearful” to “sad” while preserving audio and text locality.Figure 6 shows the edit also updating reasoning to a new context.Source: J. Case Study · p. 34, Figure 6 · p. 35
  • Sequential MEND editing on DeSTA can produce repetitive, incoherent outputs and failures across reliability, generality, locality, and portability.Figure 7 illustrates the degeneration case.Source: J. Case Study · p. 34, Figure 7 · p. 36
  • Sequential-edit degeneration patterns vary: MEND is severe on DeSTA, KE on all models, and UnKE on AF.The frequencies are summarized in Table 18 and are consistent with Figure 4 trends.Source: J. Case Study · p. 34
Loading 2510.16917v2…