Source-linked AI summary
StrixAE: An Intelligent Agent for Audio Enhancement under Complex Distortion Coupling in Real-World Scenarios
Chenglin Wu, Junjie Wu, Jinhang Chen, Mingyang Chen, Zixu Lin, Jiabian Chen, Xinghao Ding, Xiaotong Tu
TL;DR
Real-world audio enhancement requires handling coupled distortions and personalized requirements beyond existing methods. StrixAE uses an MLLM controller trained with CoT supervision and structured audio-perception reinforcement learning, and experiments report performance exceeding most open-source and proprietary solutions with strong generalization robustness.
Problem
Existing audio-enhancement methods struggle to handle coupled distortions and personalized enhancement simultaneously in real-world scenarios.
Method
StrixAE is an MLLM-based audio agent trained with supervised fine-tuning and APRL using separate format, structure, and perceptual-quality rewards.
Results
StrixAE outperforms most open-source and proprietary solutions across real-world test datasets, with strong generalization robustness.
Takeaways & Limitations
The proposed framework provides an agent for autonomous multi-scenario audio enhancement with executable pipelines and logically ordered reasoning.
Takeaways & Limitations
The experiments use lightweight, efficient task-specific tools, and more advanced tools may further improve performance.
Abstract
from arXiv · showhide
Audio enhancement in real-world scenarios involves complex distortion couplings and requires personalized enhancement. Existing solutions struggle to address both simultaneously. To improve robustness and enable autonomous operation in such scenarios, we propose StrixAE, an agent based on a multimodal large language model (MLLM). StrixAE leverages the MLLM as a controller to coordinate multiple audio enhancement and personalization models. To further enhance system robustness, reduce artifacts, and improve generalization across diverse real-world scenarios, StrixAE is trained through a two-stage process: first, CoT supervised fine-tuning on AcoustBench to ground basic reasoning and tool invocation; second, Audio Perception Reinforcement Learning (APRL), a reward design specifically tailored for audio restoration pipelines that jointly optimizes format validity, structural coherence, and perceptual quality. Unlike generic RL fine-tuning, APRL introduces structured rewards that enforce executable pipelines and logical section ordering, enabling the agent to produce reliable, interpretable enhancement plans without hallucinated tools. Based on real-world test datasets, our proposed method outperforms most existing open-source and proprietary solutions, achieving state-of-the-art performance across multiple perceptual metrics and demonstrating strong generalization robustness.
1 Introduction
Real-world audio enhancement must handle coupled distortions and personalized requirements that exceed the scope of task-specific and all-in-one methods. StrixAE addresses this gap with an MLLM-based agent, AcoustBench, and two-stage training with structured audio-perception rewards.
- Task-specific methods target limited enhancement objectives, while all-in-one methods struggle with coupled distortions and personalized augmentation in real-world scenarios.
- Unknown distortion combinations motivate an MLLM controller that can reason over real-world degradations and coordinate expert enhancement models.
- StrixAE is an MLLM-based audio agent designed to perceive degradation factors and autonomously orchestrate expert models across multiple enhancement scenarios.
- The two-stage training paradigm combines supervised fine-tuning with APRL, whose rewards separately address format validity, structural reasoning order, and perceptual quality.
- AcoustBench contains 88.9K instruction-response pairs with explicit chain-of-thought annotations and executable tool chains for scheduling under coupled distortions.
2 Related Work
Prior audio-enhancement research is organized mainly around task-specific models, while agent research emphasizes multimodal reasoning and tool use. Reinforcement learning and AI feedback provide alignment mechanisms that reduce reliance on manual audio-quality annotation.
- Speech Enhancement: Audio-enhancement studies commonly address denoising, dereverberation, separation, restoration, or target-speaker extraction through task-specific datasets and models.
- Speech Enhancement: Real-world enhancement requirements are scenario-dependent, motivating adaptive combinations of multiple subtasks inferred from the input audio.
- LLM-Empowered Agent: Recent multimodal-agent research highlights MLLMs’ reasoning, decision-making, and tool-use capabilities in complex environments.
- Agentic Reinforcement Learning: AI-generated feedback in audio evaluation can approximate human judgments while reducing reliance on manual annotation and lowering dataset-construction costs.
3 Method
StrixAE combines an MLLM-based audio enhancement agent with AcoustBench and a two-stage training framework to handle complex degraded audio. Its reinforcement-learning stage rewards executable tool pipelines, ordered reasoning, and perceptual speech quality.
- 3.1 Overview: StrixAE coordinates multiple audio enhancement models to process complex degraded audio and produce high-quality enhancement.
- 3.2 Dataset Construction: AcoustBench is constructed as a four-stage benchmark with explicit chain-of-thought annotations and samples containing audio, clean targets, instructions, reasoning, and tool sequences.
- 3.2 Dataset Construction: The dataset uses clean, noisy, and reverberant audio resources to construct paired audio with composite distortions for diverse enhancement scenarios.
- 3.3 Training: StrixAE is trained with supervised fine-tuning followed by reinforcement learning to improve structural consistency, output validity, and perceptual awareness.
- 3.3.2 Audio Perceptual Reinforcement Learning: The reinforcement-learning reward combines format, structure, and perceptual-quality components, with positive weighting hyperparameters balancing their contributions.
- 3.3.2 Audio Perceptual Reinforcement Learning: The perceptual reward combines normalized DNSMOS and ESTOI through a weighted geometric mean to evaluate speech quality and intelligibility.
4 Experiment
StrixAE is evaluated on real-world blind datasets using multiple non-intrusive perceptual metrics and comparisons with existing systems. It achieves strong enhancement quality, generalization, and training benefits from APRL and coordinated orchestration.
- Experimental Setup: StrixAE is evaluated on AcoustBench and real-world blind test data using DNSMOS, NISQA, UTMOS, and SCOREQ.The benchmark includes public DNS and URGENT Challenge data plus the AcoustBench-Real dataset.
- Main Results: StrixAE achieves state-of-the-art DNSMOS, UTMOS, and SCOREQ results on AcoustBench-Real and its personalized enhancement test set.It outperforms most open-source methods and surpasses several closed-source baselines on some metrics.
- Main Results: 0.17, 0.43, 0.28, and 0.32 are StrixAE’s improvements over TF-GridNet across the reported metrics.These gains are reported for the AcoustBench-Real evaluation comparison.
- Generalization: StrixAE narrows performance gaps to leading closed-source systems across real-world and blind-test datasets, supporting strong generalization robustness.On the blind test set, reported gaps are 0.02, 0.02, and 0.21 for DNSMOS, NISQA, and SCOREQ.
- Qualitative Analysis: StrixAE recovers speech frequencies, improves denoising, and better preserves harmonic structures than the compared single-task and integrated models.These qualitative differences are shown through spectrum visualization on the Real-Recording test dataset.
- Ablation and Orchestration: StrixAE-Orchestrated outperforms its SFT-based counterpart in most scenarios, while jointly learned model selection and task ordering achieve optimal speech-quality results.Table 3 reports that both learned selection and ordering outperform fully random planning, and their integration further improves StrixAE-SFT.
- Training Dynamics: APRL stabilizes format and structure rewards quickly, whereas the APQ reward improves more gradually because audio-quality optimization is more difficult.The reported dynamics indicate sequential stabilization from format and structure toward perceptual quality.
5 Ablation Study
The ablations show that both learned model selection and predicted task ordering improve performance over random planning, while their combination performs best. The two-stage SFT-plus-RL strategy outperforms either training approach alone across evaluation metrics.
- Both learned model selection and task order prediction outperform fully random planning, verifying the necessity of each module.
- Combining task planning and dynamic model routing further improves StrixAE-SFT, with StrixAE-Orchestrated achieving the best results across all speech quality metrics.
- SFT alone yields competitive performance across all metrics, whereas RL alone causes noticeable performance degradation.
- Combining SFT with RL achieves the best performance across all evaluation metrics, supporting the proposed two-stage training strategy.
6 Conclusion
The paper introduces StrixAE and a two-stage training framework with structured reinforcement rewards for reliable audio-agent operation. Experiments report superior generalization robustness and state-of-the-art performance across real-world and personalized enhancement tasks.
- StrixAE is trained with a two-stage framework that includes APRL and decomposed rewards for format validity, inference structure, and perceptual quality.
- AcoustBench is introduced as a large-scale benchmark for learning and evaluating audio-agent scheduling under complex distortion coupling.
- Across real-world datasets and personalized enhancement tasks, StrixAE outperforms most open-source and proprietary solutions.
- StrixAE achieves state-of-the-art performance on multiple perceptual metrics and exhibits superior generalization robustness.