PaperlayerView exact revision on arXiv

Source-linked AI summary

SAO-Instruct: Free-form Audio Editing using Natural Language Instructions

Michael Ungersböck, Florian Grötschla, Luca A. Lanzendörfer, June Young Yi, Changho Choi, Roger Wattenhofer

arXiv:2510.22795v1cs.SDcs.LG

TL;DR

Audio editing with natural language remains limited by methods that require full target descriptions or predefined operations. SAO-Instruct fine-tunes Stable Audio Open on triplets made through Prompt-to-Prompt, DDPM inversion, and manual edits; it generalizes to real in-the-wild audio and performs competitively on objective metrics while outperforming other approaches in a subjective listening study.

  • Problem

    Existing audio-editing approaches require complete target descriptions or constrain instruction-based edits to predefined tasks, although users may give diverse, underspecified instructions.

    Source: §1 · pp. 1–2

  • Method

    SAO-Instruct fine-tunes Stable Audio Open on input-audio, edit-instruction, and output-audio triplets created with Prompt-to-Prompt, DDPM inversion, and manual edits.

    Source: §1 · p. 2, §3.3 · p. 6, §3.4 · p. 6

  • Results

    SAO-Instruct generalizes to real in-the-wild audio and diverse edits, achieves competitive objective performance, and outperforms other audio-editing approaches in subjective listening tests.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    SAO-Instruct edits from a single free-form instruction, whereas prior baselines require both input- and target-audio captions.

    Source: §6 · p. 9

    Generating the Prompt-to-Prompt and DDPM inversion datasets is computationally expensive and constrained by the underlying generative model's capabilities.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

Generative models have made significant progress in synthesizing high-fidelity audio from short textual descriptions. However, editing existing audio using natural language has remained largely underexplored. Current approaches either require the complete description of the edited audio or are constrained to predefined edit instructions that lack flexibility. In this work, we introduce SAO-Instruct, a model based on Stable Audio Open capable of editing audio clips using any free-form natural language instruction. To train our model, we create a dataset of audio editing triplets (input audio, edit instruction, output audio) using Prompt-to-Prompt, DDPM inversion, and a manual editing pipeline. Although partially trained on synthetic data, our model generalizes well to real in-the-wild audio clips and unseen edit instructions. We demonstrate that SAO-Instruct achieves competitive performance on objective metrics and outperforms other audio editing approaches in a subjective listening study. To encourage future research, we release our code and model weights.

1 Introduction

Audio editing must selectively change specified sounds while preserving the surrounding context, but existing methods require full target descriptions or restrict users to predefined operations. SAO-Instruct introduces free-form instruction-based editing and a triplet dataset built from complementary synthetic and manual editing pipelines.

1 Introduction

  • Audio edits must selectively change specified aspects while preserving background context, even though user instructions vary in scope and structure.Existing zero-shot methods require full descriptions of edited audio, while instruction-based alternatives may be limited to predefined tasks.Source: §1 · pp. 1–2
  • SAO-Instruct is presented as the first free-form instruction-based audio editing model, trained on input-audio, instruction, and output-audio triplets.The dataset combines Prompt-to-Prompt, DDPM inversion, and manual editing.Source: §1 · p. 2
  • SAO-Instruct generalizes to real in-the-wild audio and diverse editing operations despite partial training on synthetic data.Source: §1 · p. 2
  • The authors report competitive performance against zero-shot approaches that rely on full audio descriptions.Source: §1 · p. 2

2 Related Work

Stable Audio Open is a latent-diffusion general-audio model, while generative editing spans zero-shot and supervised approaches. Existing editing methods may require detailed input and output descriptions or offer only fixed operations, motivating flexible instruction-based editing.

2.1 Text-to-Audio Generation

  • Stable Audio Open encodes stereo audio into a continuous latent representation before a diffusion transformer generates up to 47 seconds of 44.1 kHz stereo audio.The model conditions on text, timing information, and the diffusion timestep.Source: §2.1 · p. 2

2.2 Generative Editing

  • Generative editing includes zero-shot methods that adapt existing models and supervised methods fine-tuned for editing tasks.Zero-shot inversion typically needs detailed input and output descriptions, which can be unavailable or difficult to phrase in real-world settings.Source: §2.2 · p. 3
  • Supervised instruction-based systems let users specify intended changes, but AUDIT uses five core operations and other models are not tailored to high-fidelity editing with free-form instructions.Related work also includes infilling, music editing, and broader audio transformation approaches.Source: §2.2 · p. 3

3 Method

SAO-Instruct is fine-tuned on audio-editing triplets assembled through prompt generation and three complementary audio-pair creation routes. The method combines synthetic Prompt-to-Prompt data, real-input DDPM inversion samples, and deterministic manual edits.

3 Method

  • The method constructs input-audio, edit-instruction, and output-audio triplets from fully synthetic, semi-synthetic, and manually edited audio pairs, then fine-tunes Stable Audio Open.Source: §3 · pp. 3–4

3.1 Prompt Generation

  • GPT-4o generates edit instructions and output captions from AudioCaps and WavCaps input captions, along with metadata used for filtering and synthesis.For example, “Birds chirping and water flowing” can yield “Remove the water flowing” and the output caption “Birds chirping”.Source: §3.1 · p. 4

3.2 Prompt-to-Prompt

  • Prompt-to-Prompt transfers attention maps from the input prompt to edited-prompt generation, with seed and CFG search followed by Bayesian optimization of edit parameters.Candidate pairs are screened with Gemini and CLAP; the optimization objective evaluates generated-pair quality and edit accuracy.Source: §3.2 · pp. 4–5
  • Attention injection controls input-output similarity, while injection delay and token weighting tune when and how strongly changed caption tokens influence generation.An injection fraction of 1 enforces strong similarity; changed-token attention can be scaled to emphasize the proposed edits.Source: §3.2 · pp. 4–5

3.3 DDPM Inversion

  • DDPM inversion creates semi-synthetic pairs with real input audio and generated output audio, using Bayesian optimization to set source and target CFG strengths and edit flexibility.The inversion timestep trades input consistency against editing flexibility.Source: §3.3 · p. 6

3.4 Manual Edits

  • Twelve deterministic, interpretable manual operations produce edited pairs, with instructions generated from captions and optional parameters then varied or shortened probabilistically.The operations include ADD, REPLACE, DROP, SWAP, LOOP, PITCH, SPEED, LOW_PASS, HIGH_PASS, INPAINT, SUPER_RES, and DENOISE; variation and minimization are each applied with 50% probability.Source: §3.4 · pp. 6–7

4 Experimental Setup

SAO-Instruct is fine-tuned on audio-editing triplets and evaluated with both objective metrics and a participant listening study. The evaluation combines measures of audio similarity and perceptual quality with ratings of edit relevance and faithfulness.

4.1 SAO-Instruct

  • Triplets combine Prompt-to-Prompt examples from AudioCaps and WavCaps, DDPM-inversion examples from AudioSet-SL, and manual edits from a separate WavCaps subset.Final samples are stored as 44.1 kHz stereo WAV files.Source: §4.1 · p. 7
  • The model is fine-tuned from Stable Audio Open on input audio, output audio, and edit-instruction triplets, then conditions on the instruction, input audio, and its length.At inference, the input is encoded into the diffusion model’s latent space, noised, and used to start denoising.Source: §4.1 · p. 7

4.2 Evaluation Metrics

  • Objective evaluation measures distributional similarity with FD, LSD, and KL, perceptual quality with IS, and edit-caption alignment with CLAP.The authors note that CLAP filtering may bias results toward its embedding space; they avoid CLAP training loss and report multiple objective and subjective metrics.Source: §4.2 · p. 7
  • Thirteen participants rated 10-second AudioCaps test clips for quality, relevance, and faithfulness on a discrete 1–5 MOS scale.Model names were hidden and output order randomized per sample.Source: §4.2 · pp. 7–8

5 Evaluation

The evaluation compares dataset-generation variants and audio-editing baselines using objective metrics and subjective ratings. SAO-Instruct performs strongly in listening tests despite using only an edit instruction, while dataset generation has practical constraints.

5 Evaluation

  • Evaluation uses 1,000 AudioCaps test samples and both original-audio and synthetic-caption references, since similarity metrics alone may reward edits that ignore instructions.The original reference is intended to capture naturalness, while the regenerated reference serves as a proxy for edit accuracy.Source: §5 · p. 8

5.1 Ablation

  • Combined training datasets perform robustly across metrics, balancing the advantages of Prompt-to-Prompt, DDPM-inversion, and manual-edit data.Prompt-to-Prompt edits may be more precise but less faithful to input qualities, while DDPM inversion and manual edits may preserve those qualities better but be less precise.Source: §5.1 · p. 8

5.2 Comparison with Baselines

  • Nearly 8x faster than AudioEditor, SAO-Instruct outperforms other models in edit relevance and faithfulness while maintaining high audio quality.On FD, LSD, and KL, models perform similarly, with no model dominating across both reference proxies; AudioEditor has higher CLAP and Inception scores, alongside lower faithfulness.Source: Table 3 · p. 9, §5.2 · p. 9
  • The baseline comparison gives ZETA full input and target descriptions and AudioEditor descriptions plus changed-word indices, whereas SAO-Instruct receives only the edit instruction.SAO-Instruct is trained on a combined dataset of 150,000 triplets.Source: §5.2 · p. 9

5.3 Limitations

  • Prompt-to-Prompt and DDPM-inversion dataset generation is computationally expensive and constrained by the capabilities of the underlying generative model.The authors focus on general audio; applying the approach to music would require suitable generative music models and music triplets.Source: §5.3 · p. 9

6 Conclusion

The paper concludes that SAO-Instruct supports varied free-form audio edits while preserving input context, using triplets built from Prompt-to-Prompt, DDPM inversion, and manual edits. Its prompt-generation pipeline uses an LLM to create edit instructions, output captions, and filtering metadata from input captions.

6 Conclusion

  • SAO-Instruct performs a wide range of free-form edits while preserving input context, outperforming zero-shot baselines in subjective tests and achieving competitive objective results.Unlike prior baselines described here, it requires only a free-form edit instruction rather than both input and target captions.Source: §6 · p. 9

A Prompt Generation

  • The prompt-generation pipeline passes dataset captions to an LLM for edit instructions, output captions, and metadata used in filtering and synthetic-sample quality control.The LLM reasons about audible elements, and prompts with more than two distinct elements are filtered out because current generative models may omit elements or produce artifacts.Source: Figure 4 · p. 13, A Prompt Generation · p. 13
  • Optional negative captions specify audio elements to exclude, supporting synthetic generation with Prompt-to-Prompt and, for DDPM inversion, negative output prompts only.For an instruction to remove birds, for example, the negative output caption contains “birds chirp,” while the negative input caption is empty.Source: A Prompt Generation · p. 13

B Manual Edit Tasks

The manual-edit task set comprises twelve operations, with different input requirements and optional controllable parameters. Input and parameter constraints keep edits within the model’s audio-length limit.

B Manual Edit Tasks

  • The manual-edit task set contains twelve operations: ADD, REPLACE, DROP, SWAP, LOOP, PITCH, SPEED, LOW_PASS, HIGH_PASS, INPAINT, SUPER_RES, and DENOISE.The tasks vary in required inputs, input constraints, and whether they accept controllable parameters.Source: B Manual Edit Tasks · p. 13
  • Input constraints account for task input count, relative clip lengths, and distinct audio elements to keep outputs within Stable Audio Open’s 47-second limit.For example, ADD overlays two clips, HIGH_PASS processes one clip, and DROP removes only one element from a composite audio.Source: B Manual Edit Tasks · p. 14
  • ADD, LOOP, PITCH, and SPEED accept parameters, while REPLACE, DROP, SWAP, LOW_PASS, HIGH_PASS, INPAINT, SUPER_RES, and DENOISE do not.Parameter bounds include a one-octave limit in either direction for PITCH and a 47-second maximum for LOOP outputs.Source: B Manual Edit Tasks · p. 14, Table 5 · p. 14

C Dataset Generation

The supplied material reports the compute costs of generating editing triplets and describes the scope of the accompanying task and captioning evaluations.

C Dataset Generation

  • Generating 100k samples—50k from each method—used 6.2k GPU hours, while manual edits took about 5.1 seconds per sample on one CPU core.Prompt-to-Prompt candidate selection averaged 2.2 minutes per prompt pair, followed by about 3.4 minutes for sample generation; DDPM inversion took 1.8 minutes per audio pair.Source: C Dataset Generation · p. 14
  • Table 6 covers eight editing tasks that take no parameters.Its caption specifies that len(·) returns audio length in seconds and elem(·) counts audible elements.Source: Table 6 · p. 15
  • The AudioCaps test subset was used to evaluate Stable Audio Open and its fine-tuned variant.The accompanying prose reports that the fine-tuned version followed prompts more closely.Source: Table 7 · p. 15

D Fine-tuning

The model is fine-tuned by training the diffusion transformer while keeping the autoencoder frozen, and inference from a noised input latent better preserves input characteristics while retaining editing flexibility. The listening study gathered volunteer ratings of quality, relevance, and faithfulness.

D Fine-tuning

  • Fine-tuning keeps the autoencoder frozen and trains only the diffusion transformer; the AudioCaps-fine-tuned model followed prompts more closely.The AudioCaps run used 47k training and validation samples for 15 epochs; the final combined 150k-sample model trained for 80 hours on two NVIDIA A6000 GPUs.Source: D Fine-tuning · p. 15

E Inference

  • Starting from the Gaussian-noised encoded input latent preserves more input characteristics while retaining enough flexibility to perform edits.The prose also reports higher CLAP and lower FD/KL relative to the target caption-conditioned regenerated reference.Source: E Inference · p. 16, Table 8 · p. 16

F Listening Study

  • Thirteen unpaid volunteers rated edited audio for quality, relevance to the instruction, and faithfulness to the input on five-point scales.The endpoints ranged from poor quality to same quality as input, completely irrelevant to perfect instruction following, and completely different from same as input.Source: F Listening Study · p. 16

G.1 Comparison with Baselines

Against zero-shot editing baselines, SAO-Instruct slightly underperformed ZETA in production quality and slightly outperformed it in production complexity. The deterministic manual-editing evaluation used signal- and frequency-based metrics across tasks with unambiguous targets.

G.1 Comparison with Baselines

  • SAO-Instruct slightly underperformed ZETA in production quality but slightly outperformed it in production complexity.The authors report that these objective-metric results reflect the subjective listening study; Table 9 compares the two measures.Source: §G.1 · p. 16
  • The subjective listening study compared SAO-Instruct with audio-editing baselines using the interface shown in Figure 5.The supplied passage identifies the interface but does not report listening-study outcomes.Source: Figure 5 · p. 17

G.2 Deterministic Editing

  • Deterministic manual-editing evaluation used 100 AudioCaps test samples per edit type and excluded ADD and REPLACE because their target audio was ambiguous.Metrics included STFT, MR-STFT, MR-MEL, SI-SDR, and SI-SNR; the passage describes averages in Table 10 and per-task results in Table 11 without giving their values.Source: §G.2 · p. 17

G.3 Qualitative Results

Qualitative examples show both global and localized edits, while the authors also describe failure modes and broader-impact safeguards. They report using licensed, established datasets and models and call for further work on responsible deployment and detection of edited audio.

G.3 Qualitative Results

  • Examples include adding ambient noise to speech without distorting the primary signal, reducing rain volume, and replacing crumpling paper with typing while preserving speech.The model performs these edits from the input audio and free-form instruction alone, without access to the original or target captions.Source: §G.3 · pp. 17–18, Figure 6 · p. 18
  • Edits can depend on instruction phrasing, fail to blend added sounds into the background, or alter the wrong sounds in complex scenes.The model may also struggle to reconstruct coherent speech or produce audible artifacts; the authors attribute limitations primarily to insufficient data diversity and suggest larger, more diverse training data as mitigation.Source: §G.3 · p. 18, Figure 7 · p. 20, Figure 8 · p. 20

H Broader Impacts

  • The authors identify deceptive or harmful manipulation as a misuse risk and say they use well-established datasets rather than scraping arbitrary sources.They encourage research on responsible deployment practices and detection of synthetically edited audio.Source: H Broader Impacts · p. 18

I Licenses

  • The authors state that they comply with the licenses and terms for the datasets, models, and other assets used.The passage names the MIT, CC BY 4.0, and CC0 1.0 licenses among those assets and notes the Stable Audio Open research-use terms.Source: I Licenses · p. 20

Related papers

Loading 2510.22795v1…