Source-linked AI summary
MRMAD: A Multi-Round Multi-Audio Benchmark for Evaluating Acoustic Degradation Perception in Large Audio-Language Models
Yize Li, Ningyuan Yang, Sile Yin, Sindhuja Thogarrati, Sung-En Chang, Andrew C. Singer, Xue Lin, Chuan-Che Huang, Shuo Zhang
TL;DR
LALMs have advanced in semantic audio understanding, but whether they reliably perceive audio degradation remains unclear. MRMAD addresses this gap with a multi-round, multi-audio benchmark across domains and tasks, finding substantial limitations in current degradation-aware perception and reasoning.
Problem
Whether LALMs can reliably perceive low-level audio-quality differences beyond high-level semantic cues remains underexplored.
Method
MRMAD evaluates 18 LALMs across speech, music, and sound using multi-round tasks for degradation identification, severity comparison, and severity ranking.
Results
Current models show substantial limitations, while Gemini 3.1 Pro achieves 86.22% on DTI, 90.81% on DSC, and 64.25% on DSR.
Takeaways & Limitations
Degradation-aware perception is a critical missing capability for audio-language modeling.
Takeaways & Limitations
The benchmark does not propose training methods or architectural solutions, leaving identified issues for future work.
Abstract
from arXiv · showhide
Large audio-language models (LALMs) have shown promising progress in understanding speech, music, and general sound events, yet their ability to reason about how audio signals are degraded remains underexplored. Existing benchmarks primarily evaluate semantic understanding, event recognition, or high-level audio reasoning, leaving a basic question unanswered: Do LALMs understand the differences in audio quality? We introduce MRMAD, a Multi-Round Multi-Audio Degradation benchmark for evaluating audio degradation perception and understanding in LALMs. MRMAD spans speech, music, and sound, and frames evaluation as multi-turn dialogues over multiple audio inputs, requiring models to identify degradation types, compare severity, and perceive corruption changes across turns. Unlike current single-turn audio-language benchmarks, MRMAD evaluates whether LALMs can maintain consistent degradation hypotheses with new evidence and explain low-level acoustic phenomena in natural language. Through a systematic evaluation of 18 representative LALMs from non-thinking to reasoning and Omni models, we find that current models often recognize coarse content while failing to diagnose, compare, or reason about degradations reliably. MRMAD reveals an important yet overlooked aspect of audio-language understanding and provides a diagnostic foundation for building future LALMs that are robust to real-world acoustic conditions.
1 Introduction
MRMAD addresses the underexplored problem of whether LALMs can perceive and reason about low-level audio degradations rather than relying mainly on semantic cues. It introduces a multi-round, multi-audio benchmark spanning domains and tasks to diagnose this capability.
- Audio quality perception remains underexplored despite progress in semantic audio understanding, event recognition, and high-level reasoning.
- Existing benchmarks mainly assess content understanding, instruction following, domain knowledge, or reasoning, while speech-quality evaluation remains single-audio and speech-specific.
- Its multi-round, multi-audio format supplies one clip and prompt per round, enabling comparison through conversational context without concatenating clips.
- MRMAD evaluates audio quality understanding across speech, music, and sound using nine degradation types and three tasks: DTI, DSC, and DSR.
- The benchmark contains 8,400 multiple-choice questions and evaluates 18 LALMs for degradation identification, severity comparison, and severity ranking.
2 Related Work
Prior audio-language benchmarks emphasize semantic understanding, instruction following, and reasoning across auditory domains. MRMAD extends this evaluation focus toward low-level audio-quality perception in multi-turn, multi-audio settings.
- Large Audio-Language Models: Recent LALMs and LARMs support broad auditory interaction, including speech, music, sound, long-audio understanding, and complex reasoning.
- Benchmark Comparison: MRMAD differs by explicitly comparing benchmark size, domain coverage, multi-turn design, and evaluation focus against existing audio-language benchmarks.
- Audio-Language Benchmarks: Existing benchmarks evaluate diverse audio understanding and reasoning capabilities, including chat-based tasks, speech understanding, scene understanding, and paralinguistic voice understanding.
3 MRMAD Benchmark
MRMAD constructs a balanced benchmark from speech, music, and sound clips by simulating domain-specific degradations at three severities and generating multi-turn tasks. Its tasks identify degradation types, compare severity, and rank degraded clips.
- Overview: MRMAD contains 8,400 questions spanning speech, music, and sound and covering nine degradation types.
- Tasks: DTI identifies a degradation against a clean reference, DSC compares severity between two degraded clips, and DSR ranks three degraded clips.
- Source Audio Data: Source clips are selected for high quality and domain diversity so introduced degradations remain perceptually distinguishable.
- Source Audio Data: The dataset includes 5,400 speech clips, 4,800 music clips, and 4,200 sound clips, with audio converted to mono and resampled to 16 kHz.
- Degradation Simulation: Each reference is RMS-normalized, degraded at audible, medium, and strong severity levels, and adjusted across domains for perceptual comparability.
- Question & Prompt Sampling: Questions use multi-round prompts, varied natural-language templates, answer-only constraints, and balanced sampling across tasks, speakers, tracks, clips, and degradation types.
4 Experimental Setup
The experiments evaluate open- and closed-source audio-language models with a common multi-turn, multiple-choice protocol. Models receive audio, prompts, and options, then return one parsed answer label for comparison.
- Models: The evaluation includes 18 recent open-source and closed-source models using their official inference configurations.
- Evaluation Scheme: MRMAD uses an output-based multiple-choice protocol in which each model receives multi-round audio inputs, text prompts, and candidate options.
- Evaluation Scheme: Table 4 reports performance across three tasks and audio domains, with average scores computed as correct predictions divided by total samples.
- Evaluation Scheme: All models use identical multi-turn inference templates and answer parsing for fair comparison.
5 Results and Discussions
MRMAD results show that current LALMs struggle with degradation-aware perception and reasoning, with performance varying by task, model family, degradation type, reasoning setup, and audio representation. Diagnostic analyses link these failures to weak reference use, inconsistent cross-turn grounding, and suppression of degradation cues in projected audio tokens.
- Main Results: Most models remain near random baselines on DTI and DSR, while several are near the 50% DSC binary-choice baseline.This indicates persistent difficulty capturing low-level degradation cues despite progress in general audio understanding and reasoning.
- Main Results: 86.22% on DTI, 90.81% on DSC, and 64.25% on DSR make Gemini 3.1 Pro the best model across all three tasks.It improves over the best open-source model by 49.64, 30.11, and 30.67 absolute points, respectively; GPT-Audio-1.5 performs poorly, especially on DSC.
- Main Results: Qwen3-Omni-Thinking leads open-source DTI and DSC with 36.58% and 60.70%, while Qwen2.5-Omni records the strongest open-source DSR result at 33.58%.MiMo-Audio-Instruct is the strongest non-omni LALM across most task averages, but open-source systems remain far behind Gemini 3 families.
- Reasoning vs. Non-Reasoning: Reasoning gains are mixed: Qwen3-Omni-Thinking improves by 1.27, 8.06, and 6.50 points on DTI, DSC, and DSR, while matched-pair averages change by −0.25, +0.93, and −0.79 points.Other reasoning variants underperform their non-reasoning counterparts, showing that reasoning-oriented training alone does not guarantee robustness.
- Degradation Type Analysis: DTI is more degradation-dependent than DSC: stronger models perform better on packet loss and low-pass filtering but worse on MP3 compression, band-pass filtering, and reverberation.The pattern suggests greater sensitivity to coarse temporal or broad high-frequency cues than to subtle spectral, spatial, or artifact-specific distortions.
- Why do LALMs Fail on MRMAD?: Removing or corrupting the clean first-turn reference causes only limited drops, and several models achieve comparable or higher accuracy without the first turn.These results indicate under-utilized reference audio and difficulty retaining or comparing evidence across turns, with extra context sometimes adding misalignment or memory burden.
- Why do LALMs Fail on MRMAD?: Projected audio tokens can suppress degradation information: SALMONN maintains high clean-degraded similarity, whereas Audio Flamingo 3 shows moderate sensitivity and Qwen2.5-Omni lower, degradation-dependent similarity.Lower similarity indicates greater sensitivity to degradation-induced acoustic changes; high similarity suggests invariance to low-level quality distortions.
6 Conclusion
MRMAD evaluates degradation-aware auditory understanding across audio domains through DTI, DSC, and DSR. The benchmark shows limitations in fine-grained perception and multi-audio severity reasoning.
- MRMAD evaluates degradation-aware auditory understanding across speech, music, and sound using degradation type identification, severity comparison, and severity ranking.
- Current models remain limited in fine-grained degradation perception and multi-audio severity reasoning.
- The reported limitations involve unstable reasoning gains, degradation-dependent behavior, weak cross-turn grounding, and degradation-insensitive audio representations.
Limitations
MRMAD is an evaluation benchmark rather than a training or architectural solution. Its analyses identify limitations, but addressing them is left to future work.
- MRMAD does not propose new training methods or architectural solutions for improving LALMs.
- Addressing identified limitations in degradation-aware perception, cross-turn grounding, and audio-token representations is left to future work.
- The study cannot fully disentangle whether closed-source model gains arise from audio modeling, broader multimodal capabilities, scale, or proprietary training procedures.
- Human performance is not reported because expert human annotators were limited, with expert evaluation planned for future work.
Ethics Statement
MRMAD uses publicly available audio datasets and simulated degradations for perceptual degradation understanding. Its stated scope excludes speaker identification, private-attribute inference, and high-stakes decisions.
- MRMAD is constructed from publicly available audio datasets and simulated degradations.
- The benchmark evaluates audio-language models on perceptual degradation understanding rather than speaker identification, private-attribute inference, or high-stakes decisions.
- The audio sources are used in accordance with their original licenses and usage terms.
- The benchmark may reveal weaknesses in current LALMs but does not introduce methods for generating harmful content or manipulating audio.
A.3 Band-Pass Filtering
The appendix describes simulated degradations and the controls used to define their severity. Band-pass filtering narrows retained frequency content while holding the affected frequency region fixed across severity levels.
- Band-Pass Filtering: Band-pass filtering preserves a frequency band around a sampled center frequency while attenuating frequencies outside the passband.
- Band-Pass Filtering: Smaller bandwidth fractions retain narrower frequency ranges and produce stronger degradation.
- Band-Pass Filtering: For each reference clip, the center frequency remains fixed across severity levels, while only bandwidth varies to isolate bandwidth restriction.
- Band-Pass Filtering: Speech and music use progressively smaller bandwidth fractions from audible to strong severity, while sound events use larger fractions to preserve more spectral content.
- Other simulated degradations include MP3 compression, reverberation, echo, neural-vocoder artifacts, DSP denoising artifacts, and packet loss.
- Packet-loss severity increases through more frequent and longer loss bursts, with lost 30 ms packets replaced by zeros.
B Additional Analysis
The additional analysis finds severity ranking hardest because it requires preserving severity-sensitive evidence across multiple clips and conversational turns. Performance depends on more than model size, including degradation-sensitive representations and acoustic grounding.
- DSR is the most difficult task, with the lowest scores for nearly all models.The best open-source model reaches only 33.58%, slightly above the 25% random baseline.
- Ranking multiple degraded clips requires preserving severity-sensitive acoustic evidence across multiple audio inputs and conversational turns.
- Model size alone does not explain MRMAD performance.MiMo-Audio-Instruct remains competitive with 7B parameters, while some larger or similarly scaled models perform worse.
- MRMAD performance depends on degradation-sensitive audio representations, training scale, and grounding language outputs in acoustic evidence.
- The two-turn template carries the first audio, prompt, and response into a second turn before presenting the next audio and prompt.The three-turn template extends this procedure with another audio-text turn.
D Token-level Similarity
Token-level analysis shows that projected representations can preserve high-level content while losing degradation-specific evidence. Audio Flamingo 3 encodes severity more clearly, but its separation remains imperfect and non-uniform.
- SALMONN’s projected tokens show high similarity across strongly degraded audio and degradation types, attenuating degradation-specific differences.
- Severity information may be compressed or lost after projection, limiting evidence for downstream severity reasoning.
- Audio Flamingo 3 shows a clearer monotonic trend in which stronger degradations generally correspond to lower token similarity.
- Audio Flamingo 3’s severity separation remains imperfect and varies across token positions, so severity is only partially and non-uniformly encoded.
- Improving degradation-aware LALMs requires preserving fine-grained spectral, temporal, and degradation-specific cues in audio encoders and projection modules.
E Response Examples
Response examples show that success depends on aligning intermediate descriptions with perceptual cues and maintaining those cues across turns. Failures arise from lost task grounding, ambiguous artifacts, inconsistent global ranking, and sensitivity to prompt design.
- Response examples: Correct DTI responses connect lost high-frequency detail and muffled timbre to low-pass filtering.
- Response examples: Correct DSC responses compare clips along the same degradation dimension and identify the clip with the stronger perceptual artifact.
- Response examples: Qwen2-Audio can lose task grounding, while Qwen3-Omni-Thinking may disagree with annotated severity for ambiguous neuralvocoder artifacts.
- Response examples: Gemini 3.1 Pro can describe individual clips correctly yet fail to maintain a consistent global ranking or map it to the correct option.
- Prompt design: Diversified, degradation-specific prompt templates reduce reliance on repeated wording, positional bias, generic quality differences, and textual priors.
- Prompt design: DSR extends DSC from two clips to three and uses adjacent-swap distractors to test fine-grained severity reasoning.