Source-linked AI summary
SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning
Jiajun Fan, Jingyuan Li, Prashanth Gurunath Shivakumar, Jia-Hong Huang, Qi Luo, M. Maruf, Ivan Bulyko, Ge Liu, Roger Ren
TL;DR
Voice agents need native-speech training for multi-turn tool use, but existing approaches either attach speech around text training or evaluate through costly, non-differentiable APIs. SpeechGym provides a local audio-native RL environment with fixed agentic tasks and trainable feedback, and its trained agent more than doubles success on an independent voice benchmark while improving efficiency and behaviour.
Problem
Existing voice-agent approaches either train in text or use proprietary audio APIs that block gradients and make on-policy reinforcement learning prohibitively costly.
Method
SpeechGym has local omni-modal user and agent models converse in native audio over unmodified tasks, tools, databases, policies, and success checks, with per-turn rewards for tool-call outcomes.
Results
Pass@1 more than doubles from 24% to 53% without further tuning on an independently implemented voice benchmark, with gains across all three domains.
Takeaways & Limitations
SpeechGym indicates that audio-agentic failures are trainable perceptual and behavioural competences rather than a fixed cost of speech modality.
Takeaways & Limitations
Deployment risks beyond the simulated gym remain out of scope and require safeguards such as content filtering, consent, and human escalation.
Abstract
from arXiv · showhide
Voice agents must call tools and hold multi-turn dialogue entirely through speech, yet the dominant paradigm trains them in text. Existing frameworks either cascade TTS and ASR around a proprietary voice API, where gradients cannot flow and per-call cost makes on-policy reinforcement learning prohibitive, or stay in text: they measure voice agents but cannot improve them. We present SpeechGym, an audio-native agentic environment in which two omni-modal models converse in native audio, with no external ASR or TTS and no API boundary, over the unmodified tasks, tools and success check of an established text agentic benchmark, so that the interaction modality is the only variable and the loop stays local and trainable end to end. Audio agentic capability does not follow from audio understanding. The failures speech introduces are perceptual rather than reasoning deficits: the agent picks the right tool and the right argument slot but fills it with a value misheard from the waveform, and that single error cascades into a failed call, a retry of the same call, and a wasted step budget. A second failure is behavioural: under an insistent caller the agent performs an unauthorised write and ends the episode believing it helped. Both are trainable, because the environment labels them for free: a call with a misheard argument fails against the database while a correct one succeeds. The obstacle is sparsity, not signal. Outcome-only GRPO is gradient-starved here, since almost every rollout group fails identically, while a per-turn process reward crediting each successful tool call restores variance to nearly every group. Trained this way, the agent transfers with no further tuning to an independently implemented voice benchmark, more than doubling task success and carrying an open-weights model from last place to second on that leaderboard, while using fewer turns and tokens than before training.
1 Introduction
SpeechGym makes speech the only changed modality in a locally trainable agentic loop, exposing perceptual and behavioural failures that text-based or evaluation-only voice systems cannot improve. Per-turn rewards address sparse feedback, enabling transfer to an independent voice benchmark.
- SpeechGym: SpeechGym holds tasks, tools, databases, policies, and success checks fixed while changing only the user–agent channel from text to native speech.Its local loop contains the user simulator, tools, and reward, with no external ASR, TTS, or proprietary API boundary.
- SpeechGym: SpeechGym combines audio-native interaction, multi-turn tool use, and end-to-end reinforcement-learning trainability in one environment.It exposes a reset/step/reward interface for text-agentic domains and open omni-modal models, and adds Banking for high-stakes numeric slots.
- Failure diagnosis: 32% of speech rollouts mis-hear a slot value versus 2% in text, a sixteenfold increase that cascades into 42% tool errors and dead loops ending at zero reward.The dominant failures are described as perceptual rather than reasoning deficits.
- Training: Outcome-only GRPO provides gradients in only 16% of rollout groups, while per-turn process rewards raise that share to 99.6%.The process reward credits successful tool calls and penalises failed calls, restoring variance when most terminal returns are zero.
- Training: A vLLM-based rollout server makes each training epoch 5.4× faster at $0 API cost.This local setup avoids the latency and price barriers of proprietary voice APIs.
- Transfer: Pass@1 rises from 24% to 53% without further tuning on an independently implemented voice benchmark, moving an open 30B model from last to second place.The trained agent also uses fewer turns and tokens, issues fewer unauthorised writes, falls into fewer dead loops, and recovers from mis-hearings more often.
2 Related Work
Prior work separates trainable multi-turn tool use from audio interaction: text environments can support training, while voice benchmarks generally evaluate proprietary cascades without enabling end-to-end RL. SpeechGym is positioned as the first system combining native audio, tool use, and local online-RL trainability.
- Voice evaluation: τ-Voice wraps τ 2-bench in a TTS-and-ASR cascade around a proprietary realtime voice API, so it measures rather than trains voice agents.Its API boundary blocks gradients and its rollout costs rule out the volume required for on-policy RL.
- Voice evaluation: τ-Voice costs over $200 per epoch and over $1,400 for seven epochs, whereas SpeechGym runs at $0 API cost.The cost difference is presented as a practical barrier to large on-policy training runs.
- Audio models: Speech and audio models can understand or generate audio, but prior systems do not train agentic choices such as when to invoke tools, confirm values, or authorise writes.SpeechGym supplies this missing audio-agentic training signal.
- Positioning: τ-bench, τ 2-bench, and single-call tool-use RL are trainable but text-only, while prior audio frameworks are evaluation-only.Prior work therefore supplies at most two of audio-native interaction, multi-turn tool use, and online-RL trainability.
- Positioning: SpeechGym retains the benchmark’s agentic structure while placing both conversational sides inside local open models, enabling end-to-end online RL.Its design is framed as complementary to evaluation on τ-Voice’s independent pipeline.
3 SpeechGym
SpeechGym turns a text agentic benchmark into an audio-native, end-to-end trainable environment while leaving tasks, tools, databases, policies, and success checks unchanged. Its formulation separates speech perception from structured tool execution and supports GRPO training under sparse episode-level rewards.
- Environment design: SpeechGym changes only the user–agent channel from text to native speech while inheriting the benchmark’s tasks, tools, databases, policies, and success check unchanged.The environment is instantiated on τ 2-bench, but its design is not specific to that suite.
- Environment design: The environment is a partially observable setting where the hidden database is accessed through documented read and write tools, while observations arrive as user audio or textual tool results.Read tools expose records; write tools change state only when invoked with correct arguments.
- Environment design: At every step, the agent chooses between emitting a structured tool call and producing a variable-length spoken response.Tool calls execute deterministically, whereas speech conditions the frozen user model to produce another audio turn.
- Environment design: Speech introduces waveform slot extraction, prosodic pressure, and alternation between tool-call syntax and free spoken language within one context.A mis-perceived character can propagate into a tool argument, while urgency or insistence can bias action decisions.
- Environment design: Tool calls remain structured text so perceptual errors from mishearing values can be distinguished from behavioural errors such as wrong tools or unauthorised actions.This interface also lets the agent chain tool calls without consuming a conversational turn.
- Reward: The benchmark’s unchanged evaluator supplies the terminal reward, with no partial credit and R = 0 when the episode exhausts its step budget.Its reward basis can include database state, communication strings, action correctness, environment state, and natural-language assertions.
- Training: Outcome-only GRPO becomes gradient-starved when nearly every rollout group has identical failures, whereas per-turn returns create variance among failing groups and provide a learning signal.The process variant standardises per-turn returns and broadcasts them to the turn’s tokens; SpeechGym uses K = 4 rollouts per task.
4 Experiments
SpeechGym evaluates whether audio-native reinforcement learning can expose and correct speech-specific failures while preserving the underlying agentic tasks and checks. Across controlled comparisons and external transfer, process shaping improves trainability, task success, behavioral reliability, and interaction efficiency.
- 4.1 Setup: SpeechGym holds tasks, tools, databases, policies, and success checks fixed while varying only the interaction channel.The evaluation uses pass@1 under the unchanged database-and-communication criterion, and clean self-play results are diagnostic while τ-Voice transfer is the headline evaluation.
- 4.2 What breaks in speech: Speech amplifies perceptual and behavioral failures: mis-heard slot values rise from 2% to 32%, tool errors from 26% to 42%, and dead loops from 18% to 29%.The agent may select the correct tool and argument slot but insert a misheard value; unauthorized writes are also observed qualitatively.
- 4.2 What breaks in speech: A mis-hearing can cascade into a wrong argument, tool failure, repeated identical calls, and episode timeout at zero reward.This links the principal perceptual error to downstream tool and interaction failures rather than treating the outcome as an undifferentiated terminal failure.
- 4.3 Training in SpeechGym: Per-turn shaping raises the fraction of gradient-carrying groups from 16% to 99.6%, because failing rollouts differ in the number of successful tool calls.Outcome-only GRPO often produces identical all-zero groups, whereas successful and failed calls receive differentiated process feedback while the terminal check remains unchanged.
- 4.4 Cross-pipeline transfer to τ-Voice: 53% pass@1 versus 24% after training, with gains in Airline, Retail, and Telecom and non-overlapping 95% confidence intervals across domains.The trained agent transfers without further tuning to τ-Voice’s independently implemented user simulator, acoustics, and grader; Telecom rises from 4% to 24%.
- 4.4 Cross-pipeline transfer to τ-Voice: Training moves the same open-weights 30B model from last place to second on τ-Voice, while unauthorized writes fall from 23% to 10% and dead loops from 14% to 5%.Recovery after a mis-hearing rises from 42% to 62%; agent turns fall from 26 to 24 and tokens per task from 51,195 to 48,398.
- 4.5 Why it improves: mechanistic checks: RL also discovers repair behaviors such as spelling requests, corrected lookup retries, and switching lookup keys, although these strategies are not specified in the reward.The reported behavioral changes are consistent with improved completion over an unreliable speech channel.
5 Conclusion
SpeechGym reframes the text-to-audio gap as an optimization problem by enabling end-to-end speech training, diagnosing perceptual failures, and demonstrating transfer to an independent benchmark.
- SpeechGym turns the text-to-audio gap from an evaluation problem into an optimization problem with an objective, gradient, and stopping criterion.
- The dominant voice-agent failures are misheard slot values and resulting failed calls and repetition loops, rather than reasoning deficits.
- GRPO with a per-turn process reward closes much of the resulting performance gap.
- More than doubling pass@1 from 24% to 53%, the trained agent transfers without further tuning to an independently implemented benchmark.
Broader Impact
SpeechGym evaluates task completion using a synthetic database and communication outcome, while leaving manipulation-related rewards and deployment safeguards outside its scope.
- SpeechGym uses fictional users in a synthetic relational database, so no real personal information is processed during training or evaluation.
- Its reward depends on the correct final database state and information communicated, without rewarding persuasion, pressure, or manipulation.
- Deployment safeguards such as content filtering, consent for recorded or synthesised speech, and reliable human escalation remain outside the simulated gym’s scope.
- Tracking policy-relevant behaviour makes unauthorised writes measurable quantities that respond to training.