Source-linked AI summary
Advancing Speech Understanding in Speech-Aware Language Models with GRPO
Avishai Elmakies, Hagai Aronowitz, Nimrod Shabtay, Eli Schwartz, Ron Hoory, Avihu Dekel
TL;DR
Prior SALLM reinforcement-learning work has often used binary rewards or multiple-choice tasks, leaving open-format generative speech tasks less explored. This paper trains SALLMs with GRPO and text-based rewards for open-format SQA and AST, reporting improvements over SFT across metrics.
Problem
Method
The paper applies GRPO to open-format SQA and AST, using BLEU and other text-comparison metrics as reward signals for SALLM training.
Source: §1 · p. 1, §3.1 · p. 2, §6 · p. 4Results
Takeaways & Limitations
The authors conclude that GRPO with BLEU is effective for open-ended SQA and AST, while open-format tasks assess SALLMs’ generative abilities.
Source: §6 · p. 4Because SFT and GRPO differ substantially, the comparison uses each approach’s best validation-selected model for test reporting.
Source: §4.2 · p. 3
Abstract
from arXiv · showhide
In this paper, we introduce a Group Relative Policy Optimization (GRPO)-based method for training Speech-Aware Large Language Models (SALLMs) on open-format speech understanding tasks, such as Spoken Question Answering and Automatic Speech Translation. SALLMs have proven highly effective for speech understanding tasks. GRPO has recently gained traction for its efficiency in training LLMs, and prior work has explored its application to SALLMs, primarily in multiple-choice tasks. Building on this, we focus on open-format tasks that better reflect the generative abilities of the models. Our approach leverages GRPO with BLEU as the reward signal to optimize SALLMs, and we demonstrate empirically that it surpasses standard SFT across several key metrics. Finally, we explore the potential of incorporating off-policy samples within GRPO for these tasks, highlighting avenues for further improvement and further research.
1. INTRODUCTION
The introduction motivates open-format speech understanding as a way to assess SALLMs’ generative abilities and proposes GRPO-based training with text-comparison rewards. It reports gains over SFT across SQA and AST and studies off-policy samples.
1. INTRODUCTION
- Prior SALLM reinforcement-learning work often uses binary rewards that constrain open-ended answers, while unsupervised reward methods often underperform SFT.Source: §1 · p. 1
- The paper proposes training SALLMs with GRPO and BLEU or other metrics as verifiable reward signals.Source: §1 · p. 1
- The approach outperforms SFT on SQA and AST across multiple relevant metrics.The paper evaluates with BLEU, ROUGE, METEOR, and BERTScore.Source: §1 · p. 1
- The authors study whether off-policy samples can be incorporated into GRPO for open-format speech tasks.Source: §1 · p. 1
2. RELATED WORK
The related work positions SALLMs as speech-input, text-output systems and reviews reinforcement-learning approaches and text-similarity rewards. It highlights open-format evaluation and mixed-policy sampling as directions explored in this work.
2. RELATED WORK
- SALLMs take speech input and produce text, distinguishing them from speech language models that output speech.They use a speech encoder, language model, and lightweight projector to align modalities.Source: §2 · p. 1
- GRPO learns on-policy from model-generated samples and estimates advantages by normalizing grouped samples without a value model.Source: §2 · p. 1
- Prior GRPO work on SQA focused on multiple-choice benchmarks, which may not reflect generative ability in open-ended, chat-oriented QA.Source: §2 · p. 2
- Mixed-policy methods combine on-policy and off-policy samples, and may be useful when high-quality off-policy data is available.This work explores adding ground truth to the GRPO sample group for SALLMs.Source: §2 · p. 2
- Text-similarity metrics including BLEU, BERTScore, ROUGE, and METEOR have been used to evaluate outputs and as rewards in text-generation settings.This work investigates these metrics as rewards for speech-related SALLM tasks.Source: §2 · p. 2
3. METHOD
The method uses GRPO to optimize speech-model responses from sampled outputs and reward scores, with BLEU as the selected reward and a mixed-policy variant that adds the reference as an off-policy sample.
3.1. GRPO
- GRPO samples multiple responses per prompt, scores them, estimates advantages from the rewards, and optimizes a DAPO-variant loss.The loss is a slight variation of the original GRPO loss.Source: §3.1 · p. 2
3.2. Mixed-Policy GRPO
- Mixed-policy GRPO adds the ground-truth reference as one off-policy sample alongside G −1 on-policy samples, with clipping disabled for that reference.The method sets the unknown off-policy generating policy πϕ to 1 and leaves multiple references for future work.Source: §3.2 · p. 3
3.3. Reward functions
- BLEU scores generated text against a ground-truth reference on a 0-to-1 similarity scale and is used as the reward for open-ended responses.The paper focuses on BLEU and similar text-comparison rewards because of computational constraints; the table compares reward functions on Granite Speech 2B trained on LibriSQA.Source: §3.3 · p. 3, Table 3 · p. 3
4. EXPERIMENTAL SETUP
The experiments evaluate SQA and English-to-German AST, tasks with multiple valid outputs, using Granite Speech models and validation-selected comparisons between SFT and GRPO.
4.1. Tasks and datasets
- The study focuses on open-ended SQA and AST, where multiple outputs can be valid, while preliminary ASR experiments yielded only minor improvements.The authors note that further work is needed on RL/GRPO for ASR.Source: §4.1 · p. 3
- SQA uses LibriSQA part I, while AST uses CoVoST2 English-to-German speech translation directly, without an ASR step.LibriSQA has 107K training samples and about 2,500 test samples; CoVoST2 has about 220K/12K/15K train/validation/test samples.Source: §4.1 · p. 3
4.2. Models configuration
- Because SFT and GRPO differ substantially, test performance is reported for each task’s best validation-selected model as the fairest comparison.Most experiments use Granite Speech 2B; some use Granite Speech 8B to test scalability, and the models were not trained on SQA or CoVoST2.Source: §4.2 · p. 3
- GRPO used group size G = 8 and β = 0.02 for stable training; β = 0 led to divergence, while higher β reduced reward performance.Training Granite Speech 2B with GRPO took up to 24 hours, compared with far lower SFT cost; comparable SFT compute budgets showed minimal gains. Mixed-policy groups used G −1 on-policy samples and one off-policy sample.Source: §4.2 · p. 4
5. RESULTS
GRPO outperforms SFT on spoken question answering and speech translation, while mixed-policy results differ by task and BLEU performs best on average among tested reward metrics.
5.1. Spoken question answering
- On LibriSQA, GRPO beats SFT and the reported SOTA; Granite Speech 2B/8B BLEU gains over SFT are 9.8%/6% (over base: 61.8%/151%).The evaluation compares generated outputs using BERTScore-F1, BLEU, ROUGE-1/2/L, and METEOR.Source: §5.1 · p. 4, §5 · p. 4
5.2. Automatic speech translation
- On CoVoST2 English-to-German, GRPO raises 2B BLEU +8.2% vs base and +3.2% vs SFT, and 8B BLEU +8% vs base and +10.9% vs SFT.For Granite Speech 8B, SFT degrades performance while GRPO continues to provide gains.Source: §5.2 · p. 4
5.3. Mixed-Policy GRPO
- Mixed-Policy GRPO improves AST BLEU over standard GRPO but degrades SQA performance; the authors suggest task-specific training exposure may relate to this difference.They propose that off-policy samples may act as useful anchors for AST, whereas greater learning demands may make SQA off-policy training less stable.Source: §5.3 · p. 4
5.4. Different rewards options
- Optimizing a reward metric usually gives the best result on that metric at the cost of others, while BLEU yields the highest average performance among tested rewards.Source: §5.4 · p. 4
6. CONCLUSION
The paper applies GRPO with BLEU to open-ended SQA and AST and reports improvements across metrics for 2B and 8B models. Mixed-policy GRPO shows promise but warrants further investigation.
6. CONCLUSION
- The proposed GRPO-and-BLEU method targets open-ended SQA and AST, which the authors describe as more effective than multiple-choice tasks for assessing SALLMs' generative abilities.Source: §6 · p. 4
- The approach improves across metrics, matches or outperforms SFT, and shows improvements for both 2B and 8B models.Source: §6 · p. 4
- Mixed-policy GRPO combines on-policy and off-policy samples, but the authors call for further investigation and testing across speech-understanding tasks.Source: §6 · p. 4