Source-linked AI summary
EXAM$^2$: $\underline{Ex}tending$ $\underline{A}udio$ $Understanding$ $in$ $\underline{M}ultilingual$ $and$ $\underline{M}ultimodal$ $Analysis$
Jiawen Wang, Xiaoxue Gao, Zi Haur Pang, Nancy F. Chen
TL;DR
Existing audio-understanding evaluations have limited language, modality, and visual coverage. EXAM2 addresses these gaps with a six-language benchmark spanning heterogeneous audio and visual answer choices, then evaluates existing models and a lightweight fusion model. Results show substantial multilingual and cross-modal performance gaps, while Gemma3n-EXAM2 reports improvements over a baseline.
Problem
Existing evaluations are largely constrained to English, narrow audio domains, and audio-only settings, limiting coverage of multilingual and cross-modal audio understanding.
Method
EXAM2 builds a unified multiple-choice benchmark combining speech, sound, music, mixed-audio scenarios, visual answer choices, and six languages.
Results
Gemma3n-EXAM2 achieves up to +21.65% average multilingual improvement compared to the baseline, while multimodal conditioning particularly enhances non-speech sound understanding.
Takeaways & Limitations
EXAM2 provides a challenging evaluation of multilingual and multimodal audio understanding across LALMs, LLMs, MLLMs, and cascaded systems.
Takeaways & Limitations
The evaluation prioritizes accuracy over efficiency, uses only a subset of AudioMCQ for visual training representations, and covers mainly European and Asian languages.
Abstract
from arXiv · showhide
Recent large audio language models (LALMs) have achieved impressive progress in audio understanding. However, existing evaluations remain largely constrained to English and narrow audio domains. Prior benchmarks typically focus on a single audio modality, i.e., speech, sound, or music, limiting the systematic investigation into how these models generalize across diverse visual scenarios. In this paper, we introduce EXAM$^2$, a benchmark for multilingual and multimodal audio understanding spanning six languages and multiple modalities, including speech, sound, music, mixed-audio settings, and visual images. By incorporating visual information alongside heterogeneous audio inputs, EXAM$^2$ enables more realistic evaluation of scene-aware audio reasoning and cross-modal comprehension. EXAM$^2$ comprises $5,667$ multiple-choice questions, $22,614$ image instances, and $135,684$ multilingual translations. We evaluate state-of-the-art open-source and proprietary LALMs as well as multimodal LLMs, revealing substantial performance gaps in multilingual and cross-modal understanding. Furthermore, we propose Gemma3n-EXAM$^2$, a lightweight fusion-model fine-tuned on EXAM$^2$-train, achieves up to $12.4\%$ improvement in multilingual settings and $21.7\%$ gains in multimodal evaluation over a strong baseline. Empirical results establish EXAM$^2$ as a challenging benchmark and pioneer future multilingual and multimodal audio intelligence research.
1 Introduction
EXAM2 addresses the limited language, modality, and visual coverage of existing audio-understanding evaluations with a unified multilingual and multimodal benchmark. It evaluates reasoning across six languages and heterogeneous audio settings, and introduces a lightweight model with reported gains over a baseline.
- Existing benchmarks are largely audio-only and provide limited investigation of interactions between heterogeneous audio sources and visual grounding.
- EXAM2 combines speech, sound, music, mixed-audio scenarios, and visual representations across English, German, Spanish, Japanese, Malay, and Chinese.
- The benchmark evaluates multilingual reasoning, semantic grounding, and cross-modal comprehension in LALMs, MLLMs, and cascaded LLM systems.
- Gemma3n-EXAM2 achieves up to +21.65% average multilingual improvement compared to the baseline.
- EXAM2 contains 5,667 MCQs, 22,614 image instances, and 135,684 multilingual translated instances.
2 Related Work
Prior work has developed audio-language models and benchmarks across speech, sound, and music, but existing evaluations provide limited multilingual coverage and visual grounding for audio understanding.
- Large Audio Language Models: LALMs integrate audio encoders with LLMs to support instruction following and complex audio understanding.
- Large Audio Language Models: General-purpose omni-models have shown strong cross-modal generalization despite not being designed specifically for audio tasks.
- Multilingual and Multimodal Audio Understanding Benchmarks: Domain-specific benchmarks separately target speech, environmental sound, and music understanding.
- Multilingual and Multimodal Audio Understanding Benchmarks: Broader benchmarks combine multiple audio domains and evaluate advanced auditory reasoning capabilities.
- Multilingual and Multimodal Audio Understanding Benchmarks: Existing benchmarks remain largely audio-only, while multilingual evaluation and multimodal grounding remain underexplored.
3 EXAM2 Benchmark
EXAM2 constructs a multilingual and multimodal MCQ benchmark from diverse audio corpora, translated questions, and visual answer choices, with filtering and expert validation throughout the pipeline.
- Overview: EXAM2 supports six languages and contains 5,667 audio instances, 135,684 multilingual language instances, and 22,614 visual answer choices.
- Overview: EXAM2-train includes 4,669 questions across four audio domains, while EXAM2-test contains 998 questions across speech, sound, and music.
- Dataset Construction: The benchmark construction pipeline collects speech, music, and environmental-sound corpora, prioritizing real-world recordings over synthetic data.
- Dataset Construction: Manual inspection removes low-quality or duplicate items, including 129 inaccessible MMAR instances, producing curated benchmark subsets.
- Dataset Construction: Visual counterparts are generated for answer choices using Stable Diffusion for smaller subsets and GPT-image-2 for EXAM2-Clotho.
- Dataset Construction: Experts review questions and generated images for semantic relevance, clarity, and alignment, replacing misleading or ambiguous visual representations.
- Dataset Construction: English questions and choices are translated into German, Spanish, Japanese, Malay, and Chinese, with terminology refinement and native-speaker review.
4 Methodology
EXAM2 formulates audio-visual multiple-choice understanding as joint reasoning over raw audio, candidate images, questions, and answer choices. Its Gemma3n-based model uses unified multimodal representations and multilingual LoRA tuning across six languages.
- Task Formulation: Each EXAM2 instance pairs a raw audio waveform, one candidate image per choice, a question, answer choices, and a ground-truth answer.The model must jointly reason over audio and visual modalities to identify the correct choice.
- Multimodal Input Representation: Gemma3n-E4B encodes audio, images, and text within a unified transformer.Audio is converted into spectrogram-derived hidden states, while each choice image is encoded into patch embeddings.
- Multimodal Fusion: The fused sequence concatenates image embeddings, audio representations, question tokens, and choice tokens without a modality-specific fusion layer.Cross-modal alignment is learned entirely through self-attention.
- Parameter-Efficient Fine-Tuning via LoRA: LoRA freezes the base model and injects trainable low-rank updates into attention and feed-forward layers, training prediction of the correct choice index.The loss is restricted to the answer token.
- Parameter-Efficient Fine-Tuning via LoRA: OmniLoRA uniformly samples English, German, Spanish, Japanese, Malay, and Chinese during training for multilingual tuning.The fused multimodal sequence is rendered in the sampled language for each training instance.
5 Evaluation
EXAM2 evaluates closed- and open-source models across three audio domains and six languages using shared prompts and zero-shot testing. Results show strong multilingual and multimodal disparities: Gemma3n-EXAM2 improves broadly, visual context often helps, but cascaded systems can suffer visual confusion and Eastern languages remain weaker.
- Evaluation Setup: Zero-shot evaluation compares closed- and open-source models across Speech, Sound, and Music using the same prompts for all languages and audio domains.Closed-source models are accessed through Azure OpenAI, while open-source models use Hugging Face evaluation.
- Overall Results: GPT-4o-audio achieves the strongest closed-source average at 73.83%, while Gemma3n-EXAM2 leads open-source models overall at 61.24%.Gemma3n-EXAM2 is strongest in Speech and Music, but Gemma3n† remains stronger in Sound, 67.12% versus 59.26%.
- Multilingual Results: +21.65% average accuracy gain across all languages accompanies Gemma3n-EXAM2, with global Sound and Music improvements of 11.91% and 13.15%.German Speech improves by 26.43%, while visual answer choices provide multimodal grounding for Sound and Music.
- Multimodal Results: +15.55% separates Qwen-2.5-omni audio+image from its audio-only variants across languages, with statistically significant improvement.Sound benefits especially strongly from visual information, gaining +11.7% for Qwen-2.5-omni and 4.31% for Gemma3n.
- Failure Analysis: Cascaded Gemma3n variants degrade when images are added alongside audio, suggesting visual information can distract late-fusion reasoning.The paper hypothesizes representational competition when visual embeddings weakly correlate with target acoustic semantics.
- Cross-Linguistic Evaluation: German, English, and Spanish generally outperform Japanese and Chinese, especially on Sound and Music tasks.Speech and Sound scores align strongly across languages, with Pearson correlation ≈ 0.80, while Sound and Music retain larger multilingual gaps.
6 Ablation Study
The ablations show that modality and language composition affect EXAM2 performance differently: multimodal inputs benefit non-speech audio, while multilingual training improves performance across all evaluated languages.
- 6.1 Ablation on Modality: 61.22% averaged across languages, multimodal conditioning outperforms audio-only at 60.05% and image-only at 41.23%.The multimodal advantage is strongest for environmental sound understanding, where visual grounding provides complementary contextual cues.
- 6.1 Ablation on Modality: Speech performs best under audio-only settings in most languages, whereas multimodal fusion offers clearer advantages for sound and smaller benefits for music.Audio remains the primary semantic signal, while visual context improves robustness and cross-domain generalization, particularly for non-speech audio.
- 6.2 Ablation on Language: Multilingual training produces substantial and consistent improvements across all six evaluation languages compared with English-only training.Under identical multimodal configurations, English-only training degrades performance by 12.4 points to 48.83% averaged across languages.
- 6.2 Ablation on Language: Multilingual training slightly improves English by +3.4 points and particularly strengthens sound and music understanding.The reported benefit therefore extends beyond non-English languages, including the dominant training language.
7 Conclusion
The paper introduces EXAM2 as a large multilingual and multimodal benchmark for evaluating audio understanding across models, domains, and languages. Results indicate that multimodal conditioning and multilingual training improve performance, while Gemma3n-EXAM2 performs competitively across the benchmark.
- 7 Conclusion: EXAM2 contains 135,684 multilingual and 22,614 multimodal instances for evaluating LALMs, LLMs, MLLMs, and cascaded systems.Evaluations span speech, sound, and music across six languages.
- 7 Conclusion: Multimodal conditioning significantly enhances audio understanding, particularly for non-speech sound tasks.The conclusion also reports substantial multilingual-training benefits across all languages, especially Eastern languages.
- 7 Conclusion: Gemma3n-EXAM2 achieves competitive performance across all languages and audio categories after training on multilingual and multimodal data.The authors present this result as evidence of the benchmark’s quality for improving multilingual and multimodal audio understanding.
- 7 Conclusion: Future work can investigate more efficient training methods and broader language coverage for EXAM2.The stated future direction follows the benchmark’s current scope and training constraints.
Limitations
The evaluation prioritizes accuracy over efficiency, uses visual representations from only a subset of AudioMCQ, and covers mainly European and Asian languages.
- Limitations: The current evaluation prioritizes accuracy over efficiency and does not incorporate inference speed or deployment constraints.Future evaluations are proposed to assess real-world model performance more comprehensively.
- Limitations: Only a subset of AudioMCQ was curated with visual representations because of limited computational resources.The authors suggest using the full dataset with more efficient training methods in future work.
- Limitations: EXAM2’s language coverage is limited to European and Asian languages.The authors propose adding broader coverage, including African language families.
Ethic Statement
The authors state that generated visual data avoids personally identifiable information and offensive or harmful content, and is used only for EXAM2 training and evaluation.
- Ethic Statement: Generated images are used solely for EXAM2 training and evaluation and are designed to exclude personally identifiable information and harmful content.The authors report taking care to avoid offensive or inappropriate material.
Use of AI Tools
The authors used GPT-5-mini to translate English questions and choices into five languages and GPT-image-2 to generate visual representations of the audio. ChatGPT also assisted with language refinement and debugging.
- GPT-5-mini translated the original English questions and choices into five other languages.
- GPT-image-2 generated visual representations of the audio.
- ChatGPT assisted with grammar, punctuation, vocabulary refinement, and debugging tasks.
A Appendix
The appendix documents multilingual translation quality control, visual-example construction, standardized multimodal prompts, and fine-tuning settings for EXAM2. It also provides figures illustrating prompts, translations, generated images, and benchmark examples.
- Multilingual Translation: Translations were reviewed by native speakers or researchers with over five years of fluency for correctness, fluency, and appropriateness.
- Multilingual Translation: Translation guidelines preserve meaning and format, use natural language, address cultural relevance, and maintain consistency across languages.
- Multilingual Examples: Reviewers kept translated answers aligned with the original answer position and preserved audio-specific words when needed.
- Figures and Examples: The appendix presents translation prompts, image-generation prompts, multimodal inference prompts, and multilingual JSON examples.
- Visual Examples: Authors manually created some evaluation images because generated images were not sufficiently relevant, while GPT-image-2 produced finer-grained images for environmental sound questions.
- Multimodal Evaluation: Multimodal evaluation uses a shared prompt, with image-only or audio-only settings created by removing the corresponding input.
- Fine-Tuning Settings: Table 6 reports Gemma3n fine-tuning hyperparameters using Gemma3n-E4B as the backbone and OmniLoRA for efficient fine-tuning.