Source-linked AI summary
WavCraft: Audio Editing and Generation with Large Language Models
Jinhua Liang, Huan Zhang, Haohe Liu, Yin Cao, Qiuqiang Kong, Xubo Liu, Wenwu Wang, Mark D. Plumbley, Huy Phan, Emmanouil Benetos
TL;DR
Prior audio agents could not use audio clips as inputs, leaving a gap for audio creation and editing from existing recordings. WavCraft describes input audio, uses an LLM to decompose requests into executable programs that call expert models, and reports stronger performance on audio generation and editing. Its case studies indicate potential for audio production, while its audio analysis and inference costs remain limitations.
Problem
Earlier audio agents could not use audio clips as inputs, limiting their range of audio-generation applications.
Source: §1 · p. 1Method
WavCraft describes input audio in natural language, combines those descriptions with user requests, and uses an LLM to decompose tasks into executable programs that call expert models.
Source: §3.1 · p. 4Results
WavCraft reports better audio-generation and editing performance than current models; on editing, its FAD is 8.97 lower than AUDIT on the add task.
Source: §1 · p. 3, §5.3 · p. 7Takeaways & Limitations
Case studies demonstrate WavCraft’s potential for audio-production applications, with the system described as transparent, interpretable, and interactive.
Source: §7 · p. 9Existing audio-analysis models limit WavCraft’s ability to describe temporal relationships in input audio, and calling diverse APIs adds inference time.
Source: §6 · p. 9
Abstract
from arXiv · showhide
We introduce WavCraft, a collective system that leverages large language models (LLMs) to connect diverse task-specific models for audio content creation and editing. Specifically, WavCraft describes the content of raw audio materials in natural language and prompts the LLM conditioned on audio descriptions and user requests. WavCraft leverages the in-context learning ability of the LLM to decomposes users' instructions into several tasks and tackle each task collaboratively with the particular module. Through task decomposition along with a set of task-specific models, WavCraft follows the input instruction to create or edit audio content with more details and rationales, facilitating user control. In addition, WavCraft is able to cooperate with users via dialogue interaction and even produce the audio content without explicit user commands. Experiments demonstrate that WavCraft yields a better performance than existing methods, especially when adjusting the local regions of audio clips. Moreover, WavCraft can follow complex instructions to edit and create audio content on the top of input recordings, facilitating audio producers in a broader range of applications. Our implementation and demos are available at this https://github.com/JinhuaLiang/WavCraft.
1 INTRODUCTION
Earlier audio agents could not use audio clips as inputs, limiting their range of applications. WavCraft combines audio descriptions, user instructions, and expert models to create or edit audio, and the paper reports better audio-generation and editing performance than current models.
1 INTRODUCTION
- Prior audio agents could not take audio clips as inputs, motivating an agent that can analyze and transform audio for broader applications.The cited discussion contrasts WavJourney’s text-driven synthesis with the need for multimodal interactive creation.Source: §1 · p. 1
- WavCraft analyzes audio references, combines their descriptions with the user query, and uses an LLM to generate executable code that invokes expert models and other modules.The system decomposes complex creation tasks into basic ones.Source: §1 · p. 2
- WavCraft supports raw-audio-based creation, modular task decomposition, multi-dialogue co-creation, and audio scriptwriting from a high-level blueprint.The paper describes scriptwriting as producing audio without explicit user instruction.Source: §1 · p. 2
- WavCraft outperforms current models on audio generation and editing, according to the reported experiments.The introduction also identifies audio scriptwriting without explicit user commands as an additional experiment.Source: §1 · p. 3
2 RELATED WORK
Related work covers language models, LLM-based agents, and audio creation and editing. It identifies limited support for non-text inputs and complex editing as gaps in existing approaches.
2 RELATED WORK
- Many multimodal language models focus on common text-guided generation, while few can accept non-text inputs, limiting practical applications such as audio editing.The section notes that some models also do not require analysis of complex user instructions.Source: §2 · p. 3
- LLM-based agents connect language models to task-specific tools, but prior audio agents described here do not accept audio clips as inputs.Examples include AudioGPT, which classifies queries into predefined tasks, and WavJourney, which generates audio from scripts.Source: §2 · p. 3
- Most existing audio generation methods synthesize from prompts rather than edit existing audio; AUDIT handles basic edits but has difficulty with combined tasks and designated local changes.The related-work discussion lists adding, removal, replacement, super-resolution, and infilling among AUDIT’s basic editing tasks.Source: §2 · p. 3
3 WAVCRAFT
WavCraft turns audio descriptions and user queries into executable programs that call specialist audio tools. Its features emphasize decomposed, controllable editing, interactive refinement, and scriptwriting from an outline.
3.1 OVERALL FRAMEWORK
- WavCraft analyzes audio clips in natural language, prompts an LLM with those descriptions and the user query to generate a script, then executes it through expert audio models.The audio analysis module may use an audio question-answering or captioning model, and the task-specific modules can be replaced with alternatives.Source: §3.1 · pp. 4–5
3.2 FEATURES
- Modular operations decompose a user instruction into basic tasks, supporting more complex editing in an explainable manner.The framework exposes a pathway for carrying out the instruction.Source: §3.2 · p. 5
- Controllable editing translates requests into executable lines that target specified attributes while leaving the rest unchanged.Source: §3.2 · p. 5
- Human-AI co-creation supports multi-round refinement and generates scripts and comments that explain the audio-creation process.The paper links these explanations to interpretability and transparency.Source: §3.2 · p. 5
- Audio scriptwriting creates a plot from a high-level outline and input audio, then uses expert models to sonify the resulting script.The paper uses this name for creating audio without explicit user guidance.Source: §3.2 · p. 5
4 TASKS
The task section describes five basic audio-editing operations and notes that the system is also evaluated on complex features. Objective-result tables cover editing tasks and AudioCaps text-to-audio generation.
4 TASKS
- WavCraft is assessed on the five basic tasks as well as complex controllable-editing and audio-scriptwriting cases.The advanced features are evaluated through case studies.Source: §4 · p. 5
- The five basic tasks are adding a clip, removing a track, replacing a track, increasing sampling resolution, and filling masked audio regions.Source: §4 · p. 5
- The objective evaluation tables cover five editing tasks and the AudioCaps evaluation set.The supplied captions identify table scope but do not provide result values.Source: Table 3 · p. 6, Table 4 · p. 6
5 EXPERIMENTS
WavCraft is evaluated on audio editing and text-to-audio generation with objective and subjective measures, followed by case studies of audio scriptwriting and human–AI co-creation. Results favor WavCraft across the reported editing and storytelling comparisons.
5.1 EXPERIMENTS SETUP
- The setup uses GPT-4 for audio programming, LTU for audio analysis, and public expert models; editing and generation are evaluated separately against task-specific baselines on AudioCaps-based data.The pipeline uses a 16 kHz sampling rate; synthesized editing samples are called ground truth in subsequent evaluations.Source: §5.1 · p. 6
5.2 EVALUATION METRICS
- Objective metrics are FAD, KL, IS, and LSD; subjective ratings use MOS, while scriptwriting is rated for relevance, coherence, naturalness, engagement, and creativity.AUDIT+ is constructed by using ChatGPT to split instructions into basic tasks and calling AUDIT recursively.Source: §5.2 · pp. 6–7
5.3 OBJECTIVE EVALUATION
- 8.97, 14.09, and 9.52 lower FAD scores than AUDIT on add, removal, and replacement, respectively; infilling scores are FAD 3.31 and LSD 1.93.Source: §5.3 · p. 7
- WavCraft achieves the best FAD and IS among three evaluated models on AudioCaps, with KL 1.68 close to WavJourney.Source: §5.3 · p. 7
5.4 SUBJECTIVE EVALUATION
- WavCraft’s editing MOS is close to ground truth and exceeds AUDIT by a large margin; it also leads frequency and time control over both, and volume MOS over AUDIT.The authors hypothesize that less perceptually significant raw ground-truth material partly explains the frequency and time results.Source: Figure 2 · p. 7, §5.4 · p. 7, Figure 3 · p. 7
- WavCraft scores better than AUDIT+ on audio-text relevance, coherence, naturalness, engagement, and creativity.Source: Figure 4 · p. 7, §5.4 · p. 7
5.5 CASE STUDY
- In human–AI co-creation, WavCraft carries prior instructions forward across successive replacement, removal, and added-cheers requests, providing executable code and comments alongside each output.Source: §5.5 · p. 8
- In audio scriptwriting, WavCraft analyzes two raw clips, writes a Python-style script conditioned on the instruction and audio descriptions, and mixes outputs from activated modules without an explicit command.Source: Figure 5 · p. 8, §5.5 · p. 8
6 LIMITATIONS
WavCraft’s audio analysis can imprecisely describe temporal relationships in input audio, and its multi-API inference adds time costs.
6 LIMITATIONS
- Existing audio analysis models limit WavCraft’s ability to describe temporal relationships in raw audio precisely.The limitation concerns temporal relationships in the input materials.Source: §6 · p. 9
- Calling diverse APIs to solve complex tasks introduces inference time costs.Reducing inference time could support more seamless human–AI co-creation and practical applications.Source: §6 · p. 9
7 CONCLUSION
WavCraft integrates an LLM with audio expert models to create content conditioned on recordings and user queries. Case studies indicate potential for audio production applications and feasibility for transparent, interpretable, interactive audio AIGC.
7 CONCLUSION
- WavCraft integrates an LLM with diverse audio expert models to create audio conditioned on input clips and user queries.The system is presented as an agent for audio content creation.Source: §7 · p. 9
- WavCraft decomposes intricate editing into basic audio tasks, then assembles their outputs under instructions from the audio programming module.The decomposition follows comprehension of users’ queries and the content of recordings.Source: §7 · p. 9
- Case studies in real-world scenarios demonstrate WavCraft’s potential in audio production applications and feasibility for transparent, interpretable, interactive audio AIGC.Source: §7 · p. 9
A PROMPT TEMPLATE.
The prompt templates use in-context examples to allocate audio modules and retain instructions across dialogue rounds. The examples show multi-step editing and generation operations applied to input recordings.
A PROMPT TEMPLATE.
- In-context examples guide the LLM to allocate audio modules, while follow-up instructions keep current requests aligned with previous dialogue instructions.Tables 5 and 6 respectively list the initial and follow-up instruction templates.Source: A PROMPT TEMPLATE · p. 13, Table 5 · p. 14, Table 6 · p. 16
- Examples combine sound extraction, filtering, volume adjustment, repetition, and mixing to modify recordings, including isolating and extending a train sound.The examples also show extracting hammer and baby-crying sounds, truncating audio, and adding reverb.Source: A PROMPT TEMPLATE · p. 15
- Other examples combine inpainting, music generation conditioned on a melody, sound extraction, and timed mixing to create a layered audio output.One instruction places rain from the first recording into generated jazz beginning three seconds in.Source: A PROMPT TEMPLATE · p. 16
- Examples also remove wind, generate saxophone and speech, simulate a room, or replace speech with barking before upsampling a mixture.A separate example separates child speech and drums, adjusts their volumes, and removes machine sound.Source: A PROMPT TEMPLATE · pp. 14–16