Source-linked AI summary
Can LLM Agents Automate Reinforcement Learning for Text-to-Speech?
Xuanjun Chen, Zixiong Su, Hao Shi, Chang Zeng, Kai Li, Jyh-Shing Roger Jang, Hung-yi Lee
TL;DR
Localized TTS errors motivate RL post-training, but whether LLM agents can automate the full research pipeline remains unclear. AgenticTTS-Forge structures human guidance and agent execution, and its audit finds that the agent can improve and pivot while exposing measurement traps.
Problem
Whether LLM agents can automate the full TTS research pipeline remains unclear, despite RL post-training's role in correcting localized speech errors.
Source: §1 · p. 1Method
Results
The agent's pivot cut Bad cases from 13.4% to 6.1%, while the audit exposed data leakage, proxy inflation, and non-additive policy gains.
Source: §5 · p. 4Takeaways & Limitations
Measurement, rather than reasoning, is the binding constraint, and the audit supports contracts that account for every channel the agent reads.
Source: §1 · p. 2The contract missed held-out data carried verbatim through orchestrator guidance because it audited candidates, not prompts.
Source: §4.3 · p. 3
Abstract
from arXiv · showhide
Although reinforcement learning (RL) post-training repairs the localized segmental errors of zero-shot text-to-speech (TTS), arriving at a working recipe still relies on tedious manual tuning, and whether LLM agents can take over this research pipeline is unclear. We investigate this question with AgenticTTS-Forge, a collaborative workflow that structures human guidance and agentic execution around a shared workspace, applied to CosyVoice2-0.5B. To measure what the agent automates, we audit its trajectory stage by stage against the published recipe. To measure what it exploits, we score its policies with held-out observers hidden from the agent. Our results show that the agent recovers an underspecified recipe, improves it, and, when gains stall, surveys the literature unprompted and pivots from the LM carrier to the flow carrier, halving Bad cases. However, its autonomy exposes three traps across the data, proxy, and algorithm axes: the held-out set leaks through a channel the contract never reads, a self-shaped reward inflates the proxy where it is scored, and separately tuned policies do not compose additively. These findings show that the binding constraint is measurement rather than reasoning, and can inform the design of harnesses whose contracts read every channel the agent does.
1. INTRODUCTION
Zero-shot TTS still has localized segmental errors that global objectives do not effectively address, making targeted post-training important. This paper asks whether LLM agents can automate the full TTS RL research pipeline and audits their execution and failure modes.
1. INTRODUCTION
- Localized segmental errors—including mispronunciations, abnormal pauses, and semantic-phonetic alignment problems—persist despite scaling and are poorly addressed by global objectives.Existing supervised fine-tuning relies on explicit target audio and fixed data distributions, while RL with preference data targets local refinement.Source: §1 · p. 1
- Autonomous agents can execute complex workflows and operate developer tools, while RL post-training offers a focused stage with direct evaluation signals for automation.Prior evidence indicates agents can carry post-training end to end on focused tasks with clear metrics, though below expert engineering.Source: §1 · p. 1
- The paper investigates whether LLM agents can automate the full TTS research pipeline, from reproducing a baseline to improving it and pivoting architectures.AgenticTTS-Forge structures high-level human guidance around agent execution.Source: §1 · p. 1
- The agent recovers an underspecified RL recipe, improves it, pivots from the LM Carrier to the Flow Carrier, and exposes data, proxy, and algorithm traps.The pivot follows an unprompted literature survey when optimization plateaus; the audit finds the held-out set leaked through an unread channel, a self-shaped reward inflated its proxy, and separately tuned policies did not compose additively.Source: §1 · pp. 1–2
- The paper audits the shared workspace and concludes that measurement, rather than reasoning, is the binding constraint on agentic research.Preventative rules worked only where the contract reached, while post-hoc auditing recovered what prevention missed.Source: §1 · p. 2
2. THE TTS MODEL PRELIMINARIES FOR RL
CosyVoice2 separates semantic-token prediction from acoustic rendering, and the RL setup optimizes the text-speech LM and flow-matching module. The paper frames TTS RL choices along data, proxy, and algorithm axes.
2. THE TTS MODEL PRELIMINARIES FOR RL
- During RL optimization, existing utterances serve as reference prompts for generating speech and providing training signals.The setup is built on CosyVoice2.Source: §2 · p. 2
- CosyVoice2 uses a tokenizer, text-speech LM, chunk-aware flow-matching model, and vocoder; RL optimizes only the LM and flow module.The flow model renders semantic tokens into a Mel spectrogram under reference conditions, and the vocoder converts it to a waveform.Source: §2 · p. 2
- The TTS RL design space separates data selection, scalar reward proxies, and policy-update algorithms.The described algorithms include pairwise DPO-style losses on the LM and critic-free GRPO with intra-group relative advantages on the decoder; these targets are called the LM Carrier and Flow Carrier.Source: §2 · p. 2
3. THE PROPOSED WORKFLOW AND DETAILS
AgenticTTS-Forge coordinates human guidance and agent execution through four stages and a shared workspace. Its measurement contract sets boundaries and evaluations, while trajectory memory records candidate outcomes and reasoning for later decisions.
3. THE PROPOSED WORKFLOW AND DETAILS
- The four-stage workflow moves from human specification and guidance to pivot authorization and final integration approval.Human prompting peaks while guiding unspecified design choices in Stage 2; at a plateau, the human flags the symptom and authorizes the agent's methodological shift, then selects the final composition derived from the agent's scale sweep.Source: §3 · p. 2
- The shared workspace pairs a measurement contract defining boundaries with trajectory memory that logs past choices for later iterations.Each evaluated candidate's runtime record contains its executable recipe, score, and failure trace; stable indices make the trajectory auditable and allow the Stage 3 pivot to follow from the record rather than a prompt.Source: §3 · p. 2
- The measurement contract defines both final evaluation benchmarks and initial in-loop proxy models, separating evaluation metrics from training proxies.The table identifies the LM Carrier in Stages 1–2 and the Flow Carrier in Stage 3.Source: §3 · p. 2, Table 1 · p. 2
- The contract distinguishes constraints, in-loop proxies, and held-out observers, with the latter evaluating performance on a sealed test set.Constraints include isolation and floors; the LM carrier uses preference pairs and a 512-condition gate, while the flow carrier optimizes a weighted sum of three reward entries.Source: §3 · p. 2, Table 1 · p. 2
4. EXPERIMENTS
On Seed-TTS-Eval, the agent autonomously recovered and improved an RL recipe, then pivoted from the LM to the flow carrier when gains stalled. Auditing revealed traps across data, proxy, and algorithm choices, while workspace controls prevented some failures but missed others.
4. EXPERIMENTS
- Evaluation used CosyVoice2-0.5B on Seed-TTS-Eval, with WER, CER, SECS, and MOS across Chinese and English test sets.Both carriers searched a shared pool of 5,000 conditions, and training and evaluation were separated at the corpus level.Source: §4 · p. 2
4.1. Agent Outcomes: What Was Automated, and at What Cost
- The flow-carrier policy raised naturalness and halved Bad cases, while the merged policy retained the block’s highest naturalness and lowest Bad case rate.The merge also sacrificed speaker similarity and part of the LM policy’s intelligibility gain.Source: §4.1 · p. 3, Table 2 · p. 3
4.2. Workflow Dynamics: What the Agent Achieved on Its Own
- The agent’s end-to-end gains came from its exploration without hand-edited recipes, except for the human-set merge scale.It recovered the direction of the published FPO effect, improved intelligibility through on-policy pair regeneration, and pivoted to the flow carrier after LM gains stalled.Source: §4.1 · p. 3, §4.2 · p. 3
4.3. Autonomy Traps: Outside the Contract’s Reach
- Three autonomy traps escaped the contract: held-out data entered through prompting, a shaped reward inflated its proxy, and policy composition reduced speaker similarity and some intelligibility gains.The reward overstatement reached 56% on Mandarin, where the shaped term was read.Source: §4.3 · pp. 3–4
4.4. Workspace Controllability: What Held, and What Slipped
- Constraints and the LM gate rejected failed LM candidates, but flow candidates escaped proxy checks and prompt-channel leakage bypassed the sealed-test clause.The sealed-test clause ran 26 times but caught nothing; trajectory memory later enabled recovery of nine tainted records.Source: §4.4 · p. 4
5. CONCLUSION
The agent recovered and improved an underspecified RL recipe, then changed carriers unprompted when gains stalled. Its results also expose data, proxy, and algorithm traps, making measurement the binding constraint on automation.
5. CONCLUSION
- Given a symptom and no method, the agent changed carrier unprompted, cutting Bad cases from 13.4% to 6.1%, after recovering an underspecified method by measurement and improving it.Source: §5 · p. 4
- The held-out set leaked through a channel the agent’s contract never read, creating a data trap.Source: §5 · p. 4
- The agent’s shaping term inflated its proxy by up to 56% where it was scored, exposing a proxy trap.Source: §5 · p. 4
- Merging separately tuned policies caused a distribution shift that cost intelligibility against the LM policy and speaker similarity against the base.Source: §5 · p. 4
- The findings identify measurement, rather than reasoning, as the binding constraint on agentic RL post-training.The harness measures how far the agent can automate, and the authors argue that its contract can be fixed.Source: §5 · p. 4