PaperlayerView exact revision on arXiv

Source-linked AI summary

WavJourney: Compositional Audio Creation with Large Language Models

Xubo Liu, Zhongkai Zhu, Haohe Liu, Yi Yuan, Meng Cui, Qiushi Huang, Jinhua Liang, Yin Cao, Qiuqiang Kong, Mark D. Plumbley, Wenwu Wang

arXiv:2307.14335v2cs.SDcs.AIcs.MMeess.AS

TL;DR

Existing audio generation often targets task-specific conditions, leaving controllable compositions of speech, music, and sound effects as a challenge. WavJourney uses LLM-generated audio scripts and compiled programs to coordinate audio models. The authors report strong benchmark results and positive subjective evaluations for multi-genre storytelling.

  • Problem

    Existing audio generation models are often limited to task-specific conditions, while real-world audio creation calls for controllable compositions of speech, music, and sound effects.

    Source: §1 · p. 2

  • Method

    WavJourney uses LLMs to create structured audio scripts and compiles them into programs that invoke task-specific audio models and operations.

    Source: §3 · p. 4, §3.1 · p. 4

  • Results

    WavJourney outperforms previous state-of-the-art methods in subjective and objective evaluations on AudioCaps and Clotho, and receives positive subjective scores on a multi-genre storytelling benchmark.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors present WavJourney as a way to create audio storytelling from text and support interactive human-machine co-creation.

    Source: §1 · p. 3, §7 · p. 9

    WavJourney's script-and-compiler design is inflexible to extend, re-mixing may produce audio unlike real-world distributions, and using multiple models adds generation time.

    Source: §6 · p. 9

Abstract

from arXiv · show

Despite breakthroughs in audio generation models, their capabilities are often confined to domain-specific conditions such as speech transcriptions and audio captions. However, real-world audio creation aims to generate harmonious audio containing various elements such as speech, music, and sound effects with controllable conditions, which is challenging to address using existing audio generation systems. We present WavJourney, a novel framework that leverages Large Language Models (LLMs) to connect various audio models for audio creation. WavJourney allows users to create storytelling audio content with diverse audio elements simply from textual descriptions. Specifically, given a text instruction, WavJourney first prompts LLMs to generate an audio script that serves as a structured semantic representation of audio elements. The audio script is then converted into a computer program, where each line of the program calls a task-specific audio generation model or computational operation function. The computer program is then executed to obtain a compositional and interpretable solution for audio creation. Experimental results suggest that WavJourney is capable of synthesizing realistic audio aligned with textually-described semantic, spatial and temporal conditions, achieving state-of-the-art results on text-to-audio generation benchmarks. Additionally, we introduce a new multi-genre story benchmark. Subjective evaluations demonstrate the potential of WavJourney in crafting engaging storytelling audio content from text. We further demonstrate that WavJourney can facilitate human-machine co-creation in multi-round dialogues. To foster future research, the code and synthesized audio are available at: https://audio-agi.github.io/WavJourney_demopage/.

1 INTRODUCTION

WavJourney addresses the gap between task-specific audio generation and controllable storytelling audio composed of speech, music, and sound effects. It uses LLM-generated scripts and a compiled program to coordinate audio models, with reported results on text-to-audio benchmarks and multi-genre storytelling.

1 INTRODUCTION

  • Existing audio models are often limited to task-specific conditions, while compositional storytelling requires controllable speech, music, and sound effects.The paper also identifies challenges in designing storylines, composing spatio-temporal audio elements, and supporting interactive, interpretable creation.Source: §1 · pp. 1–2
  • WavJourney turns textual instructions into structured audio scripts, then compiles and executes them using task-specific audio models and operations.The script represents elements and their spatio-temporal relationships, while the program provides an interpretable compositional solution.Source: Figure 1 · p. 2, §1 · p. 3
  • WavJourney is presented as contextually adaptive, compositional, training-free, and interactive, with scripts and programs supporting producer engagement.The authors contrast its composable design with end-to-end generation, which they say often fails to align audio with complex textual descriptions.Source: §1 · p. 3
  • WavJourney outperforms previous state-of-the-art methods in subjective and objective evaluations on AudioCaps and Clotho.In subjective tests, its synthesized AudioCaps audio also outperforms the ground-truth audio, which the authors describe as suggesting potential to produce realistic, contextually relevant audio from text.Source: §1 · p. 3
  • On a new multi-genre storytelling benchmark, subjective evaluations report positive scores and performance above state-of-the-art text-to-audio synthesis models.The benchmark uses metrics including engagement, creativity, and emotional resonance.Source: §1 · p. 3
  • WavJourney is also demonstrated in multi-round dialogues for interactive audio creation and human-machine co-creation.The paper frames this as relevant to audio production applications.Source: §1 · p. 3

2 RELATED WORK

Prior work applies LLMs to multi-modal tasks and generative models to specific audio domains. The paper positions compositional audio creation as less explored and notes that data-driven approaches require substantial data and training resources.

2 RELATED WORK

  • LLMs have been used as agents that decompose tasks, generate programs, and coordinate domain-specific models across vision, language, and speech.The related work describes audio systems that connect multiple foundation models for tasks including speech, music, sound understanding, and generation.Source: §2 · p. 3
  • Audio generation models commonly target a particular condition or content type rather than controllable compositions of speech, music, and sound effects.The paper identifies audio content creation as a relatively unexplored use of LLMs.Source: §2 · p. 4
  • Data-driven compositional audio creation requires sophisticated audio data with text annotations and training powerful audio models.The related work also describes traditional audio creation as involving content design, music composition, engineering, and synthesis.Source: §2 · p. 4

3 WAVJOURNEY

WavJourney converts language instructions into audio through an LLM-written script and a compiled program that invokes expert generation models and audio operations. Its script structures acoustic elements and relationships for subsequent composition.

3 WAVJOURNEY

  • WavJourney comprises an LLM-based audio script writer, a script compiler, and expert audio generation models; the pipeline generates a script, compiles it, and executes the program.The listed model types include zero-shot text-to-speech, text-to-music, and text-to-audio generation.Source: §3 · p. 4

3.1 AUDIO SCRIPT GENERATION

  • The LLM expands an abstract instruction into JSON nodes describing decomposed speech, music, and sound effects, including acoustic attributes such as duration and volume.Each node represents an audio element, allowing the scene to be composed from individual acoustic components and their spatio-temporal relationships.Source: §3.1 · p. 4
  • Foreground elements are concatenated sequentially, while background audio overlaps foreground elements; speech is foreground-only, and music and sound effects may use either layout.Each node describes an audio component; volume is assigned to each, duration to music and sound effects, and a character attribute to speech.Source: §3.1 · pp. 4–5
  • WavJourney assigns characters to described voice presets using zero-shot TTS, with presets selected for characteristics and application scenarios.The LLM is used to allocate a suitable preset to each scripted character.Source: §3.1 · p. 5

3.2 SCRIPT COMPILING AND PROGRAM EXECUTION

  • The compiler turns the script into code whose lines call generation models, audio I/O, or operations such as mixing and concatenation, then executes it.The authors say this compiler can mitigate workflow exceptions arising from LLM instability, while the textual script is intended to aid producers without programming expertise.Source: §3.2 · p. 5

4 AUDIO STORYTELLING BENCHMARK

The paper introduces a five-genre story benchmark and a subjective evaluation protocol designed to assess audio storytelling beyond conventional speech-quality measures. Stories are rated on five dimensions using a 1–5 scale.

4 AUDIO STORYTELLING BENCHMARK

  • The benchmark covers education, radio play, romantic drama, science fiction, and travel exploration, with ten ChatGPT-generated story titles per genre.Each title ranges from 15 to 25 words, and the stories are intended to evaluate WavJourney's audio storytelling capabilities.Source: §4 · p. 5, Table 1 · p. 6
  • The subjective protocol rates engagement, creativity, relevance, emotional resonance, and pace and tempo on a 1–5 scale.Its design draws on audio and product experts and is intended to assess narrative complexity, music and sound design, and vocal delivery beyond conventional Mean Opinion Score evaluation.Source: §4 · p. 5, Table 2 · p. 6

5 EXPERIMENTS

Experiments compare WavJourney with audio-generation baselines on text-to-audio benchmarks and multi-genre storytelling, and illustrate user-directed changes through multi-round dialogue. Results favor WavJourney across key comparisons, while ground-truth audio remains preferred on Clotho.

5.1 EXPERIMENTAL SETUP

  • Setup pairs GPT-4 with MusicGen, AudioGen, Bark, and VoiceFixer, then evaluates AudioCaps and Clotho generation and 30–60-second stories.Text-to-audio evaluation includes objective and subjective methods; story creation is assessed with subjective evaluation.Source: §5.1 · pp. 6–7

5.2 RESULTS ON TEXT-TO-AUDIO GENERATION

  • WavJourney outperforms AudioGen and AudioLDM in AudioCaps subjective tests and leads objective and subjective metrics on Clotho.On AudioCaps, its OVL is 3.75 versus ground truth’s 3.73; on Clotho, ground truth is preferred in 63.7% of cases.Source: §5.2 · p. 7, Table 3 · p. 8

5.3 RESULTS ON AUDIO STORYTELLING CREATION

  • On the multi-genre storytelling benchmark, WavJourney scores positively on four subjective aspects and beats AudioLDM and AudioGen across evaluated aspects.The aspects are engagement, creativity, relevance, and emotional resonance.Source: Table 4 · p. 8, §5.3 · p. 8

5.4 HUMAN-MACHINE CO-CREATION

  • Multi-round dialogue lets users add audio elements, modify acoustic attributes, and control speech content in radio-play stories.Examples include adding background barking and changing a bleat’s duration.Source: §5.4 · p. 9

6 LIMITATIONS

The authors identify constraints in extensibility, realism of compositional audio, and generation efficiency.

6 LIMITATIONS

  • Structured scripts and script compilers make WavJourney inflexible to extend with additional functional capabilities.The authors identify this as an extensibility limitation.Source: §6 · p. 9
  • Decomposing and remixing audio may produce synthetic sounds that depart from real-world distributions, particularly for music.Realistic multi-track music requires alignment of beats, chord progressions, and melody.Source: §6 · p. 9
  • Using LLMs and multiple audio-generation models adds time costs when generating complex audio content.The authors identify improved efficiency as important for practical applications.Source: §6 · p. 9

7 CONCLUSION

WavJourney connects LLMs with expert audio models to create audio from language instructions, and experiments suggest it can generate realistic caption-aligned audio and engaging storytelling content.

7 CONCLUSION

  • WavJourney schedules expert audio models to compose realistic audio by decomposing scenes into elements and their acoustic relationships.Experiments also suggest potential for engaging storytelling audio from text.Source: §7 · p. 9

A.1 DETAILS OF TEXT-TO-AUDIO GENERATION EVALUATION

The evaluation appendix describes the AudioCaps and Clotho datasets, baseline reproduction settings, and procedures for subjective evaluation.

A.1 DETAILS OF TEXT-TO-AUDIO GENERATION EVALUATION

  • AudioCaps comprises 50 725 10-second clips from AudioSet, with one training caption and five captions per validation and test clip.The retrieved dataset contains 49 274/49 837 training clips, 494/495 validation clips, and 929/975 test clips.Source: §A.1 · p. 13
  • Clotho v2 has 3839, 1045, and 1045 clips in its development, validation, and test splits, respectively, with five captions per clip.Clips last 15 to 30 seconds; captions are 8 to 20 words.Source: §A.1 · p. 13
  • AudioGen and AudioLDM baselines were reproduced on both test sets, using 929 AudioCaps samples and 1045 Clotho clips per model.AudioCaps samples were 10 seconds; Clotho lengths were randomly selected from 15 to 30 seconds, and the first caption was used as input.Source: §A.1 · p. 13
  • Subjective evaluations used at least 10 raters per sample, with system scores averaged across raters and audio samples.Workers needed a 60% average approval rate and at least 50 approved tasks; the institutional ethics committee gave a favorable opinion.Source: §A.1 · p. 13

A.2 PROMPT TEMPLATES

The appendix provides prompt templates for generating audio scripts and parsing voices.

A.2 PROMPT TEMPLATES

  • Table 5 presents the prompt template used to generate audio scripts.Source: Table 5 · p. 14
  • Table 6 presents the prompt template used for voice parsing.Source: Table 6 · p. 15

A.3 EXAMPLE OF AUDIO SCRIPT

The example audio script represents a Mars news segment as a list of music, speech, and sound-effect entries with foreground or background layouts.

A.3 EXAMPLE OF AUDIO SCRIPT

  • The example starts with dramatic orchestral music, then pairs a news-anchor welcome with background music.Source: §A.3 · p. 15
  • A foreground transition swoosh separates the welcome from a reporter’s speech in the example script.Source: §A.3 · p. 15
  • The example closes with a news-anchor sign-off over foreground orchestral outro music.Source: §A.3 · p. 15

A.4 SCRIPT COMPILER

The script compiler generates foreground and background audio separately, then aligns and mixes them into a final track. The appendix evaluates WavJourney’s text-to-audio results, realism, component choices, compiler design, inference cost, and speech-content controls.

A.4 SCRIPT COMPILER

  • The compiler separates foreground and background nodes, generates foreground segments and background audio to match their boundaries, then concatenates and offset-mixes tracks into the final output.Background audio is mixed with the foreground at offsets determined by the foreground segments.Source: §A.4 · pp. 15–16

A.5 ABLATION STUDIES ON TEXT-TO-AUDIO GENERATION

  • Across AudioCaps and Clotho splits, WavJourney outperformed AudioLDM and AudioGen on OVL and REL; AudioCaps speech-inclusive scores were 3.93 OVL and 3.90 REL, above Ground Truth (3.77 OVL; 3.85 REL).The authors attribute non-speech gains over AudioGen to LLM decomposition and additional speech-inclusive gains to text-to-speech synthesis.Source: §A.5 · p. 17, Table 7 · p. 17, Table 8 · p. 18

A.6 TURING TEST ON TEXT-TO-AUDIO GENERATION

  • WavJourney had the highest perceived-as-real rates in Turing tests—53.8% on AudioCaps and 54.0% on Clotho—but remained below Ground Truth rates of 65.8% and 62.2%.It also had the lowest fake and ambiguous rates among the compared systems; the authors note generated speech may be easier to distinguish.Source: §A.6 · p. 18, Table 9 · p. 18, Table 10 · p. 18

A.7 SYSTEM-LEVEL ABLATION STUDIES

  • System-level ablations compare open-source LLM script writing and GPT-4-generated compiler code to assess module design effectiveness and efficiency.Source: §A.7 · p. 18

A.7.1 ON OPEN-SOURCE LLMS FOR SCRIPT WRITING

  • Llama2-70B-Chat achieved 10% compilation success, rising to 54% after manual cleanup, while GPT-4 achieved 94%.The authors associate Llama2’s low initial rate with hallucinated contexts and identify instruction tuning as a possible additional effort for open-source models.Source: §A.7.1 · p. 19, Table 11 · p. 19

A.7.2 ON LLM-BASED SCRIPT COMPILER

  • The GPT-4-based compiler had a 56% execution error rate and 63.16-second average compiling time, versus no execution errors and 0.03 seconds for the hand-crafted compiler.Source: §A.7.2 · pp. 19–20, Table 12 · p. 20

A.8 THE ANALYSIS OF INFERENCE COST

  • Inference-cost analysis timed WavJourney and AudioGen on 20 AudioCaps captions for 10-second clips; the authors leave resource optimization, such as parallel inference, for future work.Testing used a single NVIDIA GeForce RTX 2080 Ti GPU.Source: §A.8 · p. 20, Table 14 · p. 20

A.9 HUMAN-MACHINE CO-CREATION (SPEECH CONTENT CONTROL)

  • A radio-play case study demonstrates multi-turn control of speech by topic, specific sentences, and conversation language.Source: §A.9 · p. 21
Loading 2307.14335v2…