PaperlayerView exact revision on arXiv

Source-linked AI summary

AudioGenie: A Training-Free Multi-Agent Framework for Diverse Multimodality-to-Multiaudio Generation

Yan Rong, Jinting Wang, Guangzhi Lei, Shan Yang, Li Liu

arXiv:2505.22053v2cs.SDcs.MAcs.MMeess.AS

TL;DR

MM2MA lacks a unified approach for diverse inputs and audio types amid paired-data scarcity and challenges in fine-grained understanding, model selection, and self-correction. AudioGenie combines generation and supervision teams with decomposition and iterative refinement, and the authors report that it outperforms state-of-the-art methods on various tasks. The paper also builds MA-Bench, a 198-video benchmark.

  • Problem

    Paired-data scarcity and challenges in multimodal understanding, handling diverse audio events, and self-correction leave a need for unified MM2MA generation.

    Source: §1 · pp. 1–2

  • Method

    AudioGenie is a training-free multi-agent system that combines generation and supervisor teams with task decomposition, adaptive expert collaboration, and iterative refinement.

    Source: §1 · p. 2

  • Results

    The authors report that objective and subjective experiments on MA-Bench show AudioGenie outperforming state-of-the-art methods on various tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MA-Bench provides a resource for MM2MA research, with 198 curated videos, event annotations, multi-type generated audio, and ground-truth audio.

    Source: §1 · p. 2

    Future work will focus on automatic evaluation metrics for quality, accuracy, alignment, and aesthetics, and real-time adaptive strategies incorporating user feedback.

    Source: §5 · p. 8

Abstract

from arXiv · show

Multimodality-to-Multiaudio (MM2MA) generation faces significant challenges in synthesizing diverse and contextually aligned audio types (e.g., sound effects, speech, music, and songs) from multimodal inputs (e.g., video, text, images), owing to the scarcity of high-quality paired datasets and the lack of robust multi-task learning frameworks. Recently, multi-agent system shows great potential in tackling the above issues. However, directly applying it to MM2MA task presents three critical challenges: (1) inadequate fine-grained understanding of multimodal inputs (especially for video), (2) the inability of single models to handle diverse audio events, and (3) the absence of self-correction mechanisms for reliable outputs. To this end, we propose AudioGenie, a novel training-free multi-agent system featuring a dual-layer architecture with a generation team and a supervisor team. For the generation team, a fine-grained task decomposition and an adaptive Mixture-of-Experts (MoE) collaborative entity are designed for detailed comprehensive multimodal understanding and dynamic model selection, and a trial-and-error iterative refinement module is designed for self-correction. The supervisor team ensures temporal-spatial consistency and verifies outputs through feedback loops. Moreover, we build MA-Bench, the first benchmark for MM2MA tasks, comprising 198 annotated videos with multi-type audios. Experiments demonstrate that our AudioGenie achieves state-of-the-art (SOTA) or comparable performance across 9 metrics in 8 tasks. User study further validates the effectiveness of our method in terms of quality, accuracy, alignment, and aesthetic. The project website with audio samples can be found at https://audiogenie.github.io/.

1 Introduction

MM2MA aims to generate context-tailored audio types from diverse inputs, but paired-data scarcity and limitations of existing methods leave a need for a unified approach. AudioGenie combines coordinated generation and supervision, refinement mechanisms, and a 198-video benchmark.

1 Introduction

  • Paired-data scarcity limits unified MM2MA systems, while existing methods may omit speech and songs or require computationally intensive training.The paper frames its question as how to support a broader range of audio events while addressing input understanding, model selection, and self-correction challenges.Source: §1 · pp. 1–2
  • AudioGenie uses generation and supervisor teams to monitor and correct pipeline steps, alongside fine-grained decomposition, adaptive expert collaboration, and iterative refinement.The generation approach decomposes requirements into sub-events, selects domain-specialized models, and uses tree-of-thought feedback to evaluate and revise outputs.Source: §1 · p. 2
  • MA-Bench comprises 198 manually curated videos with annotations, multi-type generated audio, and ground-truth audio for MM2MA evaluation.Annotations include audio types, event timestamps, and descriptions.Source: §1 · p. 2
  • The authors report that objective and subjective experiments on MA-Bench show AudioGenie outperforming state-of-the-art methods on various tasks.The supplied passage does not specify metric values or individual task comparisons.Source: §1 · p. 2

2 Related Work

Prior audio-generation work often specializes in limited input or output types, while unified approaches can demand large paired datasets and costly training. AudioGenie is presented as a training-free multi-agent alternative for multi-type audio from multimodal inputs.

2.1 X-to-Audio Generation

  • Specialized text-to-audio and video-to-audio models excel at particular tasks but lack flexibility for complex multimodal scenarios.Unified models such as AudioX handle text, video, and image inputs to produce sound effects and music, but require large paired datasets and costly training.Source: §2.1 · p. 3
  • AudioGenie instead proposes training-free collaborative generation of multiple audio types from multimodal inputs.Its stated contrast is with approaches relying on extensive training.Source: §2.1 · p. 3

2.2 Auto-agent System

  • Earlier audio agents coordinate expert models, while LVAS-Agent focuses on sound effects; AudioGenie targets unified, training-free multi-type generation from multimodal inputs.The paper positions its system as addressing diverse user needs while aiming for reliable outputs.Source: §2.2 · p. 3

3 Method

AudioGenie coordinates a generation team and a supervisor team across planning, model selection, synthesis, and correction. Its modules decompose audio requirements, use domain experts, and refine event outputs through feedback and branching retries.

3.1 AudioGenie: Dual-Layer Coordinated Structure

  • AudioGenie’s dual-layer structure assigns generation to one team and monitoring and feedback to a supervisor that checks planning, model choices, and audio outputs.The supervisor evaluates plan suitability, timing, and alignment; reviews model choices and event files; and collaborates on audio self-correction.Source: §3.1 · pp. 3–4

3.2 Fine-Grained Task Decomposition Module

  • The decomposition module identifies implied audio events from semantic and temporal input information and records each event’s type, description, and timestamps in structured JSON.For visual input, it also prompts captions describing the overall scene for subsequent style planning.Source: §3.2 · p. 4

3.3 Adaptive MoE Collaborative Entity

  • The adaptive MoE assigns each audio type to a domain expert, who selects suitable models and refines event plans; experts also self-refine and collaborate on a collective plan.The experts’ distinct perspectives are intended to cover blind spots a single agent might miss.Source: §3.3 · p. 4

3.4 Trial-and-Error Iterative Refinement Module

  • The trial-and-error module organizes each event’s generation as a tree of initial, generation, and feedback-based refinement nodes, allowing alternate generation paths.Supervisors evaluate quality, alignment, and aesthetics; fixable issues prompt targeted post-processing, while major deviations trigger a retry with prompt adjustments or another model.Source: §3.4 · pp. 4–5
  • Tree depth and retry branches are limited for computational efficiency; if requirements remain unmet, the system returns the best result among explored branches.The authors state that this early stopping strategy prevents infinite loops on particularly difficult tasks.Source: §3.4 · p. 5

4 Experiment

MA-Bench provides curated videos with multi-type audio tracks and event annotations. The implementation uses different models for multimodal interpretation and supervision, with an audio-language model evaluating and refining event audio.

4.1 Dataset

  • MA-Bench contains 198 professionally curated videos from online platforms, video-material websites, and synthetic generators, each with multi-type audio tracks.The benchmark spans seven video-level categories and includes diverse sound effects, background music, and human speech or singing.Source: §4.1 · p. 5

4.2 Implementation Details

  • Qwen2.5-VL-72B-Instruct handles image and video inputs and supervises stages one and two, while Audio-Reasoner evaluates and refines single-event audio in stage three.The tool library supplies audio-generation methods for different audio types.Source: §4.2 · p. 5

4.3 Evaluation Metrics

The section evaluates audio with objective measures suited to multiple audio types and human ratings of quality, accuracy, alignment, and aesthetics.

4.3.1 Objective Metrics.

  • AES evaluates multi-type audio across production quality, production complexity, content enjoyment, and content usefulness.These dimensions cover technical quality, scene richness, enjoyability, and potential reuse in creative projects.Source: §4.3.1 · p. 6
  • PAM adds an overall quality score for speech, music, and environmental sounds.Source: §4.3.1 · p. 6

4.3.2 Subjective Metrics.

  • MOS ratings assess quality, accuracy, alignment, and aesthetic appeal on a five-point scale, with at least 20 participants.Accuracy covers event omission or redundancy; alignment covers spatiotemporal and semantic consistency with the input.Source: §4.3.2 · p. 6

4.4 Comparison with SOTA Methods

AudioGenie is compared with existing approaches across unconstrained multi-audio generation and SISO tasks. It performs strongly on event completeness and alignment, with favorable results across several audio types and input settings.

4.4.1 Identical-input, Unconstrained-output Audio Generation.

  • 3.54 MOS-Accuracy and 3.99 PC exceed the other video-input methods, which score at or below 2.90 on those metrics.AudioGenie also achieves the best CE, MOS-Alignment, and MOS-Aesthetic results in this comparison.Source: §4.4.1 · p. 6
  • For text and image inputs, AudioGenie scores significantly higher on multiple metrics than methods that often produce incomplete, inaccurate, or style-mismatched audio.Source: §4.4.1 · p. 6
  • With video and text, AudioGenie decomposes events and schedules them by video timestamps; Figure 3 illustrates its planning of multiple distinct audio events.The reported examples contrast this event coverage with competing methods that often miss events or misalign audio.Source: §4.4.1 · p. 7, Figure 3 · p. 7

4.4.2 Single-input Single-output Audio Generation.

  • For speech generation against four recently proposed text-to-speech methods, the system performs better on multiple metrics, scoring 4.50 MOS-Aesthetic versus 2.83 or below for other methods.Source: §4.4.2 · p. 7

4.5 Ablation Studies

Ablations examine supervision, expert specialization, refinement, and tree-of-thought correction. The reported findings associate these components with improved evaluation metrics, planning, quality, or reliability.

4.5 Ablation Studies

  • Removing the supervisor team causes a noticeable drop across multiple evaluation metrics.Step-by-step supervision helps detect planning or generation deviations and omissions before errors accumulate.Source: §4.5 · p. 8
  • Domain-specific experts support finer content and style planning, while a single general expert struggles with complex tasks requiring diverse expertise.Source: §4.5 · p. 8
  • Intra-expert self-refinement and inter-expert collaborative refinement let agents improve outputs through introspection and teamwork.Source: §4.5 · p. 8
  • Tree-of-thought self-correction improves speech quality and aesthetic appeal; iterative refinement of per-event outputs gives greater control and reliability than single-pass generation.Source: §4.5 · p. 8

5 Conclusion

The conclusion presents AudioGenie as a training-free system for multimodal inputs and multiple audio types, alongside MA-Bench and favorable evaluation results. It identifies automatic evaluation and real-time user-feedback adaptation as future work.

5 Conclusion

  • AudioGenie combines generation and supervision teams with fine-grained decomposition, adaptive expert selection, and iterative per-event refinement.Source: §5 · p. 8
  • MA-Bench is introduced as the first MM2MA benchmark and contains 198 annotated videos with multiple audio types.The paper reports SOTA or comparable performance across several metrics, with qualitative assessments and user studies supporting quality, accuracy, alignment, and aesthetics.Source: §5 · p. 8
  • Future work includes automatic MM2MA metrics spanning quality, accuracy, alignment, and aesthetics, and real-time adaptation to user feedback.Source: §5 · p. 8

A Additional SOTA Comparisons on Short Videos

On videos shorter than 10 seconds, AudioGenie’s results differ by input setting: it leads on subjective measures for video-only input and on several quality dimensions for video+text input.

A Additional SOTA Comparisons on Short Videos

  • The short-video comparison considers both video-only and video+text inputs.Videos shorter than 10 seconds are selected for a fair comparison with FoleyCrafter.Source: A Additional SOTA Comparisons on Short… · p. 10
  • For video+text inputs, AudioGenie achieves the best results in audio quality, completeness, alignment, and aesthetics.The comparison uses the same experimental setting as Table 3.Source: A Additional SOTA Comparisons on Short… · p. 10
  • For video-only inputs, AudioGenie is slightly lower on PAM and PQ but achieves the best performance on all subjective metrics and outperforms others on PC, CE, and CU.The comparison is restricted to videos shorter than 10 seconds because FoleyCrafter supports only audio under 10 seconds.Source: A Additional SOTA Comparisons on Short… · p. 10

B Generalization of results to public datasets

On VGGSound and V2M, AudioGenie is comparable to state-of-the-art video-to-audio models for single events and captures multiple events in complex scenarios.

B Generalization of results to public datasets

C Results on Different MLLMs

On the V2A task, AudioGenie outperforms existing state-of-the-art models with GPT-4o or Gemini2.5-Pro as the agent, and the framework’s results are not tied to a specific MLLM.

C Results on Different MLLMs

  • With GPT-4o or Gemini2.5-Pro as the agent, AudioGenie outperforms existing state-of-the-art V2A models.The authors report that this result demonstrates framework scalability.Source: C Results on Different MLLMs · p. 10
  • Different MLLMs exhibit distinct advantages on various metrics, and AudioGenie is not reliant on a specific MLLM.Source: C Results on Different MLLMs · p. 10
  • Table 8 compares AudioGenie using various MLLMs with other state-of-the-art methods on the MA-Bench video-input task.Source: Table 8 · p. 11
Loading 2505.22053v2…