PaperlayerView exact revision on arXiv

Source-linked AI summary

Partially Edited Audio: Generation, Defense, and Open Problems

Lin Zhang, You Zhang, Nicholas Evans, Xin Wang, David Harwath, Abhinav Dhall, Bowen Shi, Chengzhe Sun, Siwei Lyu, Sanjeev Khudanpur

arXiv:2610.37640v1cs.SDeess.AS

TL;DR

Partially edited audio combines modified and unaltered portions, creating a survey gap between generation and defense perspectives and application-dependent judgments of harm. The paper surveys editing targets, approaches, evaluation, and defenses, concluding that trustworthy editing requires utility, contextual consistency, and accountability.

  • Problem

    Partially edited audio raises application-dependent questions of harm, while prior surveys have not connected editing utility with forensic accountability across generation and defense.

    Source: §2.3 · p. 2

  • Method

    The paper defines partially edited audio and surveys generation methods, evaluation, defense methods, resources, and open problems.

    Source: §1 · p. 1

  • Results

    The survey finds generation covers diverse audio types and attributes, while defense remains concentrated on temporally edited linguistic content in speech.

    Source: §6 · p. 4

  • Takeaways & Limitations

    Trustworthy editing requires utility, contextual consistency, and accountability through detecting, localizing, and tracing edits when trust is at stake.

    Source: §6 · p. 4

    Existing benchmarks mostly use clean, single-speaker, studio-quality recordings with single short edits, and are overwhelmingly monolingual.

    Source: §5 · p. 4

Abstract

from arXiv · show

Recent advances in generative audio models facilitate increasingly flexible and selective editing of linguistic content and paralinguistic, extralinguistic, and other attributes. This paradigm, named partially edited audio, differs from full audio generation as only portions of the real recording are edited. This technique can serve both beneficial and harmful applications. We therefore present a concise survey from both generation and defense perspectives, focusing on audio, as a contribution to the ICASSP 2027 special session entitled "Partially Edited Media: Perspectives from Synthesis and Defense." We first define partially edited audio and argue that what counts as ``fake'' or harmful is an application-dependent judgment, not a fixed property of the edit. We then survey audio editing by editing target, approaches, and evaluation, and then defense methods by detection, localization, diarization, and interpretability, summarizing representative benchmarks and toolkits. Finally, we discuss future directions and open problems.

1. INTRODUCTION

Generative audio systems can now substitute, insert, or delete selected portions of a recording, leaving edited and unaltered material together. Because these edits can serve beneficial or harmful purposes, the paper surveys generation and defense while treating harm as application-dependent.

1. INTRODUCTION

  • Selected portions of a recording can be substituted, inserted, or deleted, enabling audio editing beyond speech synthesis from text.The result combines edited and unaltered portions.Source: §1 · p. 1
  • Audio editing changes selected regions or attributes while leaving the remaining recording unchanged, unlike full generation, which produces a homogeneous model-generated signal.Targets range from linguistic content to paralinguistic and extralinguistic attributes.Source: §1 · p. 1
  • Partial edits can correct mispronunciations or support podcast post-production, but misuse can enable misinformation, fraud, extortion, or security breaches.Such manipulations may be less perceptually salient and harder to detect.Source: §1 · p. 1
  • Prior surveys address editing or fake localization separately, while no single framework connects editing utility with forensic accountability.The paper responds with a survey spanning generation and defense.Source: §1 · p. 1

2. FROM FULL GENERATION TO PARTIAL EDITING

Partially edited audio is characterized by selective modification of an existing recording, with edited and unaltered material coexisting. The paper connects this technical description to application-dependent judgments of harm and to the distinct but linked roles of generation and defense.

2.1. Definition

  • The definition uses two dimensions: how much of the signal or how many attributes are manipulated, and whether editing starts from an existing recording while preserving selected portions.The same operation may serve a legitimate purpose in one setting and alter a trust-relevant property without disclosure in another.Source: §2.1 · p. 1

2.2. Characteristics

  • Coexistence, locality, and dual intent distinguish partially edited audio and motivate treating it as a joint generation-and-defense problem.Locality can be temporal or structural, and utterance-level authenticity labels are insufficient when edited and unaltered portions coexist.Source: §2.2 · p. 2

2.3. What counts as “Fake”? Connection between generation and defense

  • Whether an edit should be flagged depends on disclosure, consent, and the relying party’s trust assumptions, not on a fixed property of the edit.For codec-based editing, whether signal-modified but content-unchanged segments count as edits depends on the application and should guide annotations and evaluation.Source: §2.3 · p. 2
  • Editing artifacts can supply defense cues, but whether detector analyses can reliably improve editing quality remains open.As artifacts become subtler, verification may increasingly depend on information embedded during generation, such as watermarks.Source: §2.3 · p. 2

3. TAXONOMY OF PARTIALLY EDITED AUDIO GENERATION

The generation taxonomy spans diverse editing targets and methods, with approaches differing in how they produce edits and integrate them into the unchanged context. Evaluation must assess both edited and unchanged portions, using measures and benchmarks beyond transcript accuracy alone.

3.1. Editing Targets – What are Edited?

  • Editing targets include linguistic content, prosody and speaking style, emotion, acoustic environments, music, general sound, and cross-modal or object-level audio.Content edits include substitution, insertion, deletion, and mispronunciation correction.Source: §3.1 · p. 2

3.2. Editing Approaches - How to edit?

  • Approaches progress from splicing and generate-then-splice to context-aware infilling and instruction-conditioned models, with integration choices shaping artifacts available to defense.Infilling conditions generation on left and right context; codec language models, diffusion, and flow-matching models are among the described methods.Source: §3.2 · p. 2

3.3. Generation-Oriented Evaluation and Resources

  • Evaluation should assess edited and unchanged portions because word error rate alone misses speaker, prosodic, and acoustic inconsistencies [4] [25].Criteria include correctness, consistency, preservation, smoothness, naturalness, controllability, diversity, and robustness; RealEdit and MMAE [25] broaden benchmark conditions and operations.Source: §3.3 · pp. 2–3

4. TAXONOMY OF DEFENSE AGAINST PARTIALLY EDITED AUDIO

Defense research remains focused on speech and mostly on temporally edited linguistic content, while the survey organizes defenses around detection, localization, diarization, and interpretability. It also reviews supporting datasets and toolkits.

4. TAXONOMY OF DEFENSE AGAINST PARTIALLY EDITED AUDIO

  • Defense has concentrated on speech and temporally edited utterances, often using generated portions embedded in bona fide speech rather than neural editing models.The survey notes that the same defense principles could in principle extend to music and general sound.Source: §4 · p. 3

4.1. Detection and Localization

  • Training data matters more than architecture for partial deepfake detection: fully generated-only training degrades on partial edits, while partial-edit training with short fake portions generalizes to both settings.Localization methods use frame-level cues, boundaries, or adjacent-frame inconsistency; recent work also targets manipulated words [32] [33] [38].Source: §4.1 · p. 3

4.2. Deepfake Diarization

  • Deepfake diarization extends localization by identifying which editing process produced each region, supporting fine-grained source tracing.Tracing remains comparatively underexplored for longer, complex recordings with multiple speakers and edits from different processes.Source: §4.2 · p. 3

4.3. Interpretability and Analyses

  • Detector analyses find reliance on edit-boundary artifacts, including at concatenation points, informing understanding of system failure modes.Artifacts at edit boundaries can unintentionally serve as detection cues.Source: §4.3 · p. 3

4.4. Defense-Oriented Evaluation and Resources

  • Datasets and challenges have expanded from early English and Mandarin partial-spoof resources to localization, multimodal, instruction-based, and long-form noisy settings.Open-source WeDefense [49] and DeepFense support reproducible research and benchmarking.Source: §4.4 · p. 3

5. FUTURE DIRECTIONS AND OPEN PROBLEMS

The paper calls for generation and defense to develop together, while making trustworthy editing accountable and better suited to realistic conditions. Generalization across domains and editing styles, and robust defense for partially edited audio, remain open problems.

Co-Development of Generation and Defense

  • New editing advances can create new defense problems, while matched editing systems can provide data for training and evaluating defenses.The paper proposes sharing samples from new systems for defense evaluation; reliable detection also factors into responsible release, as illustrated by Voicebox.Source: §5 · pp. 3–4

Proactive Protection

  • Trustworthy editing should balance intended utility, consistency with unedited context, and accountability when authenticity or manipulation must be transparent.Accountability can be supported by watermarking edits at generation or provenance records at capture; mechanisms need resilience to edits, re-encoding, and adversarial removal.Source: §5 · p. 4

Toward Realistic Scenarios

  • Current evaluations often use clean, single-speaker, studio recordings with single short edits, unlike practical audio with noise, reverberation, compression, multiple edits, and overlapping speakers.Multilingual editing and fully multi-speaker, multi-object settings remain open problems.Source: §5 · p. 4

Generalization and Domain Adaptation

  • Partially edited audio adds editing style to domain variability: detectors trained on one style may perform poorly on another, and boundary-artifact cues from splicing are mitigated by context-aware editing [38].Existing datasets also differ in attack types, languages, and recording conditions.Source: §5 · p. 4
  • Domain adaptation has received little attention for partially edited audio detection and localization, leaving robust detection an open problem.Source: §5 · p. 4

Defense Against Editing on Other Audio Types, Attributes,

Defense research is concentrated on linguistic content editing in speech, while generative systems increasingly manipulate other attributes and audio types. Detection, localization, diarization, and interpretability for those settings—and cross-modal consistency and synchronization—remain open challenges.

Defense Against Editing on Other Audio Types, Attributes,

6. CONCLUSIONS

The conclusion defines partially edited audio as a recording with both altered and unaltered portions, and treats whether an edit is fake as application-dependent. It finds a gap between increasingly broad generation capabilities and speech-focused defense, and argues that trustworthy editing requires joint progress on utility, consistency, and accountability.

6. CONCLUSIONS

  • Partially edited audio contains both unaltered and edited portions in one recording, and whether it is fake depends on the application rather than the edit alone.Source: §6 · p. 4
  • Generation increasingly covers speech, music, general sound, and multiple attribute types, while defense remains concentrated on temporally edited linguistic content in speech.Source: §6 · p. 4
  • Trustworthy editing requires utility, contextual consistency, and accountability, with co-development and proactive provenance rather than reactive detection alone; both trustworthy editing and defense remain open problems.Source: §6 · p. 4

Related papers

Loading 2610.37640v1…