AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models

1 Introduction

• Human-instruction audio editing can support flexible operations such as adding background sound effects, dropping sounds, and inpainting.The paper identifies applications including replacing background music, repairing incomplete audio, and enhancing low-sampled audio.Source: §1 · p. 2
• AUDIT is a latent diffusion model conditioned on the input audio and editing instruction, generating the edited audio.It is trained in a supervised manner on triplets of instruction, input audio, and output audio.Source: §1 · p. 2
• AUDIT uses editing instructions directly and aims to preserve input segments that do not need editing.This avoids requiring a complete description of the output audio.Source: §1 · p. 2
• AUDIT achieves state-of-the-art objective and subjective results for several audio-editing tasks.Its contributions include adding, dropping, replacement, inpainting, and super-resolution.Source: §1 · p. 2

2.1 Audio/Speech Editing

• Earlier audio and speech editing focused on speech or music, including inpainting, voice conversion, and music style transfer; general audio editing by human instructions remained underexplored.GAN-based methods also addressed specific tasks such as inpainting and super-resolution.Source: §2.1 · pp. 2–3

2.2 Diffusion-based Editing

• Zero-shot diffusion editing methods can struggle to target specific regions; masked-region approaches address this for inpainting but still require a complete editing-target description.The cited approaches include SDEdit and methods that guide sampling with an additional loss or replace unmodified regions during sampling.Source: §2.2 · p. 3
• Supervised diffusion editing trains on paired or generated data, and AUDIT applies this paradigm to triplets of instruction, input audio, and output audio for multiple instructed audio-editing tasks.Other cited examples train image-to-image diffusion models or adapters for image editing.Source: §2.2 · p. 3

2.3 Text-Guided Audio Generation

• Text-guided audio generation synthesizes audio matching text, whereas this work focuses on editing existing audio with human instructions.The section describes latent-space systems such as Make-an-Audio and AudioLDM, alongside discrete or autoregressive generation approaches.Source: §2.3 · p. 3

3.1 System Overview

• The system combines an autoencoder, text encoder, latent-space diffusion network, and vocoder to reconstruct waveforms from edited mel-spectrograms.The autoencoder uses a VAE with reconstruction, KL, and GAN losses; the text encoder is a frozen pretrained T5 model, and the vocoder is HiFi-GAN.Source: §3.1 · pp. 3–4
• The latent diffusion model predicts the output-audio latent from the input-audio latent and instruction embedding by learning to predict sampled noise.Its training objective compares the predicted noise with the sampled noise.Source: §3.1 · p. 4

3.2 Generating Triplet Training Data for Each Editing Task

• The triplet-data workflow covers adding, dropping, replacement, inpainting, and super-resolution in one editing model.Adding combines a second clip with the input; dropping uses a combined clip as input and its component as output; replacement inserts different clips in roughly the same area. Inpainting masks parts of an output clip, while super-resolution downsamples the input.Source: §3.2 · pp. 4–5

3.3 Advantages of AUDIT

• AUDIT directly conditions supervised diffusion training on input audio, allowing it to learn to preserve segments that need no edit.The input latent is concatenated with the noisy target latent, so the model sees the input audio in both training and inference.Source: §3.3 · p. 5
• The method uses human editing instructions instead of full output-audio descriptions, aligning its text input with real-world editing scenarios.The authors frame triplet training, input-audio conditioning, and instruction input as advantages over prior work.Source: §3.3 · p. 5

4.1 Dataset

• Training uses AudioCaps, AudioSet96, FSD50K-S, and ESC50 to generate approximately 0.6M triplets for five audio editing tasks.AudioSet96 contains approximately 149K pairs across 96 classes; FSD50K-S consists of clips longer than five seconds.Source: §4.1 · p. 6

4.2 Baseline Systems

• Adding, dropping, and replacement compare AUDIT with SDEdit using three denoising schedules: 3/4T, 1/2T, and 1/4T.The baselines use a separately trained text-to-audio latent diffusion model.Source: §4.2 · p. 6
• For inpainting, SDEdit baselines vary in whether they replace observable latent regions with ground truth, use rough or precise masked regions, or omit text input.The super-resolution baselines are SDEdit, SDEdit-Precise, and SDEdit-wo-Text; the task is treated as inpainting in the frequency domain.Source: §4.2 · pp. 6–7

4.3 Evaluation Metrics

• The evaluation measures audio quality and naturalness with Quality, instruction match with Relevance, and objective performance with LSD, FD, and KL.The text-to-audio generation model is evaluated with IS, FD, and KL; these metrics use PANNs-based measures where specified.Source: §4.3 · p. 7

4.4 Model Configurations

• The audio editing model is initialized from a pretrained text-to-audio model and trained for 500K steps on eight V100 GPUs with batch size two per device.The autoencoder compresses mel-spectrograms to latent representations, and the HiFi-GAN vocoder is trained for 200 epochs.Source: §4.4 · p. 7

5.1 Objective Evaluation

• AUDIT achieves the best reported objective performance on adding, dropping, and replacement, with task-specific LSD, KL, and FD results.Adding: LSD 1.35, KL 0.92, FD 21.80; dropping: LSD 1.37, KL 0.95, FD 22.40; replacement: LSD 1.37, KL 0.84, FD 21.65.Source: §5.1 · p. 7, Table 1 · p. 7, Table 2 · p. 8
• AUDIT achieves the best objective performance on inpainting (LSD 1.32, KL 0.75, FD 18.17) and super-resolution (LSD 1.48, KL 0.73, FD 18.14).For AUDIT, caption guidance does not significantly affect objective metrics in either task.Source: §5.1 · pp. 7–8
• The text-to-audio model achieves FD 20.19, KL 1.32, and IS 9.23, outperforming AudioLDM-L-Full at FD 23.31, KL 1.59, and IS 8.13.The authors present it as a strong baseline for generation-based editing methods.Source: §5.1 · p. 9, Table 6 · p. 9

5.2 Subjective Evaluation

• AUDIT outperforms baseline methods in Quality and Relevance across all five tasks; dropping scores highest, with Quality 78.1 and Relevance 81.0.The authors report larger improvements on adding, dropping, and replacement, where editing requires locating the region and preserving other regions.Source: §5.2 · p. 9, Table 7 · p. 9

5.3 Case Study

• Case studies show AUDIT adding a bell at the beginning of machine-gun audio and removing duck quacking while preserving bird whistles.The adding example is reported to edit the correct region without modifying audio that does not need editing.Source: §5.3 · p. 9, Figure 5 · p. 10, Figure 6 · p. 10

6 Conclusions

• AUDIT supports adding, dropping, replacement, inpainting, and super-resolution from human text instructions.It uses generated instruction, input-audio, and output-audio triplets to train a text-guided latent diffusion model; it requires simple instructions rather than output-audio descriptions and aims to preserve segments that do not need editing.Source: §6 · p. 10
• AUDIT achieves state-of-the-art objective and subjective performance across five audio-editing tasks.The authors identify exploring more editing tasks and achieving more precise control as future work.Source: §6 · p. 10

A Datasets and Data Processing

• All audio is sampled at 16 kHz; generated audio is 10 seconds, and dataset-specific padding or truncation produces the stated input durations.Audio is converted to 80 × 624 mel-spectrograms and compressed to latent representations of size 4 × 10 × 78.Source: A Datasets and Data Processing · p. 15

C Model Details

• The autoencoder trains for 50,000 steps, while audio editing and audio-generative diffusion models train for 500,000 steps, using AdamW.Training uses eight NVIDIA V100 GPUs; learning rates are 7.5e−5 for the autoencoder and 5e−5 for both diffusion models.Source: C Model Details · p. 16, Table 9 · p. 16

D Classifier-free Guidance

• Training masks text with some probability so the model learns a no-text condition score; classifier-free guidance combines that score with the text-conditioned score.The guidance coefficient s ≥ 1 balances sample diversity and quality.Source: D Classifier-free Guidance · pp. 16–17

• Ten randomly selected test samples from each task’s test set form the human evaluation set.Source: E Evaluation · p. 17
• Ten raters score each sample in the human evaluation.Source: E Evaluation · p. 17
• Raters assess Quality and Relevance on a 1-to-100 scale.Source: E Evaluation · p. 17