Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering

• Audio modality research has largely overlooked reinforcement-learning approaches despite progress in LLM reasoning and visual multimodal tasks.Source: §1 · p. 1
• AQA requires extracting insights from audio, inferring implicit relationships, and generating contextually relevant answers.Its combination of auditory and linguistic modalities makes it suitable for evaluating complex logical reasoning.Source: §1 · p. 2
• The report applies GRPO to Qwen2-Audio-7B-Instruct to explore reinforcement learning and deep thinking for audio question answering.Source: §1 · p. 2
• 64.5% accuracy on MMAU Test-mini establishes state-of-the-art performance for the reported AQA experiments.Source: §1 · p. 2
• With 38k post-training samples, reinforcement learning outperforms supervised fine-tuning, while explicit reasoning shows no significant AQA benefit.The report also states that LALMs remain far behind human auditory-language reasoning.Source: §1 · p. 2

Audio Question Answering

• AQA builds on audio captioning but additionally requires comprehension of complex acoustic patterns, temporal relationships, and contextual information.Source: §2 · p. 2
• The study focuses on single-audio, selection-based questions, where generating an answer is difficult but verifying options is straightforward.Source: §2 · p. 2

Multimodal Reasoning

• Recent multimodal studies have renewed interest in reinforcement learning and chain-of-thought reasoning for multi-step tasks.Source: §2 · p. 3

Large Audio Language Models

• Representative audio understanding models, including Qwen2-Audio, Audio Flamingo 2, and SALMONN, are described as SFT-based.Whether reinforcement learning can unlock their potential remains studied in this report.Source: §2 · p. 3

• The training method applies GRPO directly to the Qwen2-Audio-7B-Instruct large audio language model.Source: §3 · p. 3
• For each question, the model generates an answer-formatted response that is optimized with a reinforcement-learning objective.Source: §3 · p. 3
• Prompts compare direct answer generation with an explicit chain-of-thought format requiring reasoning inside <think> tags.Source: §3 · p. 3
• GRPO samples a group of responses and uses their average reward as the advantage baseline, avoiding an additional value-function approximation model.Source: §3 · p. 3
• The policy objective includes a Kullback-Leibler term, with current and former policies controlled by PPO hyperparameters.Source: §3 · p. 4
• Responses receive rule-based rewards for correctness and, when applicable, enclosing reasoning and answers in the required tags.Source: §3 · p. 4

Experimental Setup

• Approximately 38k AVQA audio-text training samples were evaluated on the MMAU Test-mini out-of-distribution test set.The training and test data came from different sources.Source: §4.1 · p. 4, §4 · p. 4, §4.2 · p. 4

Main Results

• GRPO with Prompt <2> achieved state-of-the-art average accuracy on the MMAU Test-mini benchmark.The comparison included direct inference, SFT, RL methods, leaderboard baselines, and recent RL studies.Source: §4.2 · p. 4, Table 2 · p. 5
• Deep-thinking methods overall outperformed classic SFT, with the top four methods using RL or chain-of-thought approaches.The reported top four were GRPO + Prompt <2>, GRPO + Prompt <3>, Audio-Reasoner, and Audio-CoT.Source: §4.2 · p. 5
• Using only 38k AVQA samples, RL achieved state-of-the-art performance on out-of-distribution tests, supporting strong generalization with limited training data.The authors relate this result to the generation-verification structure of AQA and small-sample structured-thinking findings.Source: §4.2 · p. 5
• Explicit CoT templates and <think> prompting did not outperform directly prompting the model to generate <answer> responses for AQA.The contribution of deep thinking and step-by-step reasoning remains unresolved for this task.Source: §4.2 · p. 5
• The accuracy of LoRA + Prompt <1> was 8.1% lower than that of GRPO + Prompt <2>.LoRA partially addresses small-data fine-tuning challenges, but SFT still performed worse than RL.Source: §4.2 · p. 6

• 64.5% accuracy on MMAU Test-mini establishes state-of-the-art performance for the reported AQA system.The system directly applies GRPO to Qwen2-Audio-7B-Instruct.Source: §5 · p. 6
• Reinforcement learning can be effectively applied to LALMs and audio multimodal tasks, particularly AQA.The report identifies AQA's generation-verification gap as especially relevant to this application.Source: §5 · p. 6
• Explicit reasoning has not shown significant benefits for AQA, leaving efficient use of deep or step-by-step reasoning unresolved.The report identifies integrating chain-of-thought into audio as a future research focus.Source: §5 · p. 7
• LALMs still lag far behind human auditory-language reasoning, motivating further exploration of reinforcement learning.Source: §5 · p. 7