Source-linked AI summary
Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering
Gang Li, Jizhong Liu, Heinrich Dinkel, Yadong Niu, Junbo Zhang, Jian Luan
TL;DR
Audio reinforcement learning has been underexplored despite progress in language and visual reasoning, motivating a study of AQA with LALMs. The paper applies GRPO to Qwen2-Audio-7B-Instruct and achieves state-of-the-art MMAU Test-mini accuracy of 64.5%.
Problem
Reinforcement learning has been largely overlooked for audio understanding, while more complex reasoning tasks are needed to explore LALMs beyond descriptive audio tasks.
Method
The study applies GRPO to Qwen2-Audio-7B-Instruct using customized instruction prompts and answer-based reinforcement-learning optimization.
Results
64.5% accuracy on MMAU Test-mini represents state-of-the-art performance, with reinforcement learning outperforming SFT using 38k post-training samples.
Takeaways & Limitations
GRPO can be effectively applied to LALMs and AQA without large datasets, including a model with 8.2 billion parameters.
Takeaways & Limitations
Explicit reasoning has not shown significant benefits for AQA, and efficiently leveraging deep thinking remains an open question; LALMs still lag far behind humans.
Abstract
from arXiv · showhide
Recently, reinforcement learning (RL) has been shown to greatly enhance the reasoning capabilities of large language models (LLMs), and RL-based approaches have been progressively applied to visual multimodal tasks. However, the audio modality has largely been overlooked in these developments. Thus, we conduct a series of RL explorations in audio understanding and reasoning, specifically focusing on the audio question answering (AQA) task. We leverage the group relative policy optimization (GRPO) algorithm to Qwen2-Audio-7B-Instruct, and our experiments demonstrated state-of-the-art performance on the MMAU Test-mini benchmark, achieving an accuracy rate of 64.5%. The main findings in this technical report are as follows: 1) The GRPO algorithm can be effectively applied to large audio language models (LALMs), even when the model has only 8.2B parameters; 2) With only 38k post-training samples, RL significantly outperforms supervised fine-tuning (SFT), indicating that RL-based approaches can be effective without large datasets; 3) The explicit reasoning process has not shown significant benefits for AQA tasks, and how to efficiently utilize deep thinking remains an open question for further research; 4) LALMs still lag far behind humans auditory-language reasoning, suggesting that the RL-based approaches warrant further exploration. Our project is available at https://github.com/xiaomi-research/r1-aqa and https://huggingface.co/mispeech/r1-aqa.
1 Introduction
Audio question answering extends audio understanding with linguistic reasoning, offering a complex setting for testing reinforcement learning. This report explores GRPO on LALMs and reports state-of-the-art AQA performance with limited-scale training.
- Audio modality research has largely overlooked reinforcement-learning approaches despite progress in LLM reasoning and visual multimodal tasks.
- AQA requires extracting insights from audio, inferring implicit relationships, and generating contextually relevant answers.Its combination of auditory and linguistic modalities makes it suitable for evaluating complex logical reasoning.
- The report applies GRPO to Qwen2-Audio-7B-Instruct to explore reinforcement learning and deep thinking for audio question answering.
- 64.5% accuracy on MMAU Test-mini establishes state-of-the-art performance for the reported AQA experiments.
- With 38k post-training samples, reinforcement learning outperforms supervised fine-tuning, while explicit reasoning shows no significant AQA benefit.The report also states that LALMs remain far behind human auditory-language reasoning.
2 Related Works
Related work positions AQA as a reasoning-intensive audio task and reviews emerging multimodal reasoning methods alongside predominantly SFT-based audio language models.
- Audio Question Answering: AQA builds on audio captioning but additionally requires comprehension of complex acoustic patterns, temporal relationships, and contextual information.
- Audio Question Answering: The study focuses on single-audio, selection-based questions, where generating an answer is difficult but verifying options is straightforward.
- Multimodal Reasoning: Recent multimodal studies have renewed interest in reinforcement learning and chain-of-thought reasoning for multi-step tasks.
- Large Audio Language Models: Representative audio understanding models, including Qwen2-Audio, Audio Flamingo 2, and SALMONN, are described as SFT-based.Whether reinforcement learning can unlock their potential remains studied in this report.
3 Method
The method directly applies GRPO to Qwen2-Audio-7B-Instruct with customized prompts, comparing explicit and implicit reasoning formats. GRPO optimizes sampled responses using group-relative rewards for answer correctness and output format.
- The training method applies GRPO directly to the Qwen2-Audio-7B-Instruct large audio language model.
- For each question, the model generates an answer-formatted response that is optimized with a reinforcement-learning objective.
- Prompts compare direct answer generation with an explicit chain-of-thought format requiring reasoning inside <think> tags.
- GRPO samples a group of responses and uses their average reward as the advantage baseline, avoiding an additional value-function approximation model.
- The policy objective includes a Kullback-Leibler term, with current and former policies controlled by PPO hyperparameters.
- Responses receive rule-based rewards for correctness and, when applicable, enclosing reasoning and answers in the required tags.
4 Experiments
Experiments compare full fine-tuning, LoRA, SFT, and GRPO-based RL using approximately 38k AVQA audio-text samples and out-of-distribution MMAU Test-mini evaluation. GRPO achieves state-of-the-art average accuracy, while RL-based methods outperform SFT and explicit chain-of-thought does not consistently help AQA.
- Experimental Setup: Approximately 38k AVQA audio-text training samples were evaluated on the MMAU Test-mini out-of-distribution test set.The training and test data came from different sources.
- Main Results: GRPO with Prompt <2> achieved state-of-the-art average accuracy on the MMAU Test-mini benchmark.The comparison included direct inference, SFT, RL methods, leaderboard baselines, and recent RL studies.
- Main Results: Deep-thinking methods overall outperformed classic SFT, with the top four methods using RL or chain-of-thought approaches.The reported top four were GRPO + Prompt <2>, GRPO + Prompt <3>, Audio-Reasoner, and Audio-CoT.
- Main Results: Using only 38k AVQA samples, RL achieved state-of-the-art performance on out-of-distribution tests, supporting strong generalization with limited training data.The authors relate this result to the generation-verification structure of AQA and small-sample structured-thinking findings.
- Main Results: Explicit CoT templates and <think> prompting did not outperform directly prompting the model to generate <answer> responses for AQA.The contribution of deep thinking and step-by-step reasoning remains unresolved for this task.
- Main Results: The accuracy of LoRA + Prompt <1> was 8.1% lower than that of GRPO + Prompt <2>.LoRA partially addresses small-data fine-tuning challenges, but SFT still performed worse than RL.
5 Conclusion
Direct application of GRPO to Qwen2-Audio-7B-Instruct achieves state-of-the-art MMAU Test-mini performance, supporting RL for audio-language models and AQA. However, explicit reasoning has not significantly benefited AQA, and these models still lag behind human auditory-language reasoning.
- 64.5% accuracy on MMAU Test-mini establishes state-of-the-art performance for the reported AQA system.The system directly applies GRPO to Qwen2-Audio-7B-Instruct.
- Reinforcement learning can be effectively applied to LALMs and audio multimodal tasks, particularly AQA.The report identifies AQA's generation-verification gap as especially relevant to this application.
- Explicit reasoning has not shown significant benefits for AQA, leaving efficient use of deep or step-by-step reasoning unresolved.The report identifies integrating chain-of-thought into audio as a future research focus.
- LALMs still lag far behind human auditory-language reasoning, motivating further exploration of reinforcement learning.