PaperlayerView exact revision on arXiv

Source-linked AI summary

Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction

Zeyang Song, Tianchi Liu, Tianrui Wang, Chenglin Xu, Steven Y. Guo, Haizhou Li

arXiv:2608.28970v1cs.SDcs.AI

TL;DR

Open-loop TTS can produce local prosodic defects that utterance-level metrics may not expose. LoopTTS uses an AudioLLM Judge to diagnose these defects and a Refiner to perform guided re-synthesis, supported by a 42K-example weakly annotated dataset. Human evaluation reports improved recovery quality over raw audio and practical open-loop re-generation baselines on diagnosed low-quality utterances.

  • Problem

    Open-loop TTS can produce sporadic local prosodic defects that utterance-level metrics may fail to expose, creating a need for corrective diagnosis and generation.

  • Method

    LoopTTS uses a Filter–Judge–Refiner pipeline in which an AudioLLM generates structured prosodic instructions and a Refiner performs guided re-synthesis, trained with approximately 42K weakly annotated examples.

  • Results

    Human evaluation shows improved recovery quality on Stage 2 flagged utterances compared with raw audio and practical open-loop re-generation baselines.

  • Takeaways & Limitations

    Bounded test-time diagnosis and refinement can improve selected low-quality TTS outputs, while the modular Judge can be replaced at inference time.

  • Takeaways & Limitations

    Human pipeline evaluation covers Stage 2 flagged utterances rather than the full distribution, and expressive re-synthesis may shift speaker embeddings.

Abstract

from arXiv · show

Current TTS systems typically rely on open-loop, single-pass generation and can produce sporadic local prosodic defects, such as misplaced stress, unnatural pauses, or flattened intonation, that utterance-level metrics often fail to expose. We present LoopTTS, a judge-guided Filter-Judge-Refiner framework for recovering low-quality TTS outputs diagnosed by an AudioLLM. Given an initial utterance from a base TTS model, an AudioLLM Judge identifies salient prosodic issues and generates structured refine instructions; a Refiner, our fine-grained instruction-following TTS model, then performs guided expressive re-synthesis conditioned on the initial utterance, target text, and instruction. To train the Refiner, we construct Refiner-DB, a 42K-example AudioLLM-annotated dataset with word-level prosodic weak supervision. Human evaluation on diagnosed low-quality utterances shows that LoopTTS can detect perceptually salient errors and correct them with the Refiner, outperforming raw generated audio and practical open-loop re-generation baselines in recovery quality. The Refiner also demonstrates stronger instruction-following ability for stress and pause control in targeted prosody modification.

1 Introduction

LOOPTTS addresses sporadic local prosodic defects that open-loop TTS and utterance-level metrics may miss by combining AudioLLM diagnosis with targeted re-synthesis. It contributes a closed-loop Filter–Judge–Refiner system, weakly supervised prosody data, and an instruction-following Refiner.

  • Motivation: Open-loop TTS can produce misplaced emphasis, inappropriate pauses, and flattened intonation that conventional utterance-level metrics may fail to expose.Such local defects can make otherwise intelligible speech sound unnatural.
  • LOOPTTS: The AudioLLM Judge produces structured instructions covering global attributes such as emotion, speed, and pitch, plus local stress and pause operations.The Judge also scores prosody naturalness and emotional fidelity on a 1–10 scale.
  • LOOPTTS: LOOPTTS introduces a three-stage Filter–Judge–Refiner architecture for judge-guided corrective generation.A base TTS generates an utterance, an AudioLLM diagnoses salient prosodic issues, and a Refiner performs guided expressive re-synthesis.
  • Contributions: REFINER-DB provides approximately 42K examples with word-level prosodic weak supervision for training the Refiner.The dataset supports fine-grained instruction following for flagged prosodic defects.
  • Contributions: The Refiner uses a position-aware objective that focuses supervision on instruction-specified prosodic positions.This enables guided expressive re-synthesis of diagnosed stress and pause defects.

2 Related Work

Related work establishes strong LLM-based TTS backbones and controllable speech generation, while speech editing and AudioLLM judging address complementary parts of the problem. Existing approaches generally lack fine-grained prosodic correction instructions for re-synthesis.

  • Controllable and expressive TTS: Recent LLM-based TTS systems provide a dominant backbone for high-quality speech synthesis.Controllable TTS has expanded across emotion, natural-language description, and instruction-guided synthesis.
  • Controllable and expressive TTS: Speech editing methods support posthoc content replacement or insertion but do not expose controls such as adding emphasis to a specified word.Their primary focus is content-level modification rather than fine-grained prosodic control.
  • LLM-as-Judge for TTS quality optimization: AudioLLM judges can assess fine-grained acoustic details and prosody for TTS quality optimization and preference alignment.These systems are used because human preference data is expensive and inter-annotator agreement can be low.

3 The LOOPTTS Framework

LOOPTTS is an offline cascade that filters severe failures, diagnoses retained utterances with an AudioLLM, and re-synthesizes only flagged outputs. Its Refiner is trained from contrastive AudioLLM annotations that encode global style and word-level prosodic differences.

  • 3.1 Three-Stage Quality Assurance Pipeline: LOOPTTS narrows expensive processing through coarse filtering, AudioLLM diagnosis, and Refiner re-synthesis of flagged utterances.The pipeline operates as offline quality assurance rather than streaming TTS.
  • 3.1 Three-Stage Quality Assurance Pipeline: Stage 1 uses WER and UTMOS to remove severe content or quality failures before later processing.Utterances pass when WER≤3.0% and UTMOS>3.0; failed cases receive up to two additional seeded generations.
  • 3.1 Three-Stage Quality Assurance Pipeline: Stage 2 scores prosodic naturalness and emotional fidelity on a 1–10 scale, accepting scores at least τjudge=5 and flagging lower scores for refinement.Refine instructions cover global emotion, speed, and pitch together with local stress and pause operations.
  • 3.1 Three-Stage Quality Assurance Pipeline: Stage 3 computes yR = Refiner(yinit, I, T) to perform guided expressive re-synthesis conditioned on initial audio, instruction, and target text.The output may return to Stage 2 for bounded iterative refinement, while unrequested regions and speaker embeddings can change.
  • 3.2 Contrastive Instruction Data Construction: REFINER-DB pairs expressive recordings with neutral clones of the same text and speaker, then uses AudioLLM comparisons to extract word-level stress and pause differences.Global metadata and local annotations are merged into natural-language instructions and approximately 42K training tuples.

4 Refiner: Architecture and Training

The Refiner is a fine-grained instruction-following TTS model that conditions on a refine instruction, initial utterance, and target text, using position-aware losses to target sparse prosodic edits and preserve span structure.

  • Position-Weighted Cross-Entropy Loss: Position-weighted CE amplifies supervision at instruction-specified stress and pause positions, which may represent only 10–30 of 200+ codec tokens.This concentrates learning on sparse prosodically critical regions rather than distributing gradients uniformly.
  • Architecture: The Refiner receives the refine instruction I, initial utterance yinit, and target text T, with I placed first to condition subsequent processing.The initial utterance provides an acoustic reference before the target text T.
  • Position-Weighted Cross-Entropy Loss: Prosodic spans are identified from word-level timestamps and encoded as per-token labels: 1 for stress, 2 for pause, and 0 otherwise.These spans support both position-weighted CE and structural alignment, although structural alignment uses stress spans only.
  • Position-Aware Structural Alignment Loss: The structural alignment loss compares span-level representational similarity matrices between predicted hidden representations and target acoustic-code embeddings.Each matrix entry is a cosine similarity between positions, and squared Frobenius distance is averaged across spans with length normalization.
  • Position-Aware Structural Alignment Loss: The overall objective combines position-weighted CE and structural alignment as L = LCE + λ · Lstruct, using λ=0.1 in all experiments.Pause spans are excluded from relational alignment because their silence-token structure is considered uninformative.

5 AudioLLM Evaluation

The evaluation tests whether AudioLLMs can diagnose salient prosodic defects and annotate stress and pause differences for Refiner-DB construction, using human judgments and professional labels.

  • Evaluation Setup: Five AudioLLMs are evaluated on held-out Regular and Emotional utterances for diagnosis alignment and prosodic annotation capability.The evaluation uses data sourced from EmoVoice and LibriTTS.
  • Diagnosis Capability: Human evaluators rate whether each AudioLLM diagnosis aligns with their judgment on a 1–4 scale across 100 TTS-generated utterances.The set contains 50 Regular and 50 Emotional utterances, and each model also generates refine instructions.
  • Diagnosis Capability: Gemini-3-pro achieves the highest human–AudioLLM alignment scores on both subsets: 3.3 Regular and 3.4 Emotional.It is selected as the main Judge, while its diagnoses are treated as fallible signals rather than ground truth.
  • Annotation Capability: Professional annotators and AudioLLMs label prominent stress and pause positions on 200 neutral–expressive utterance pairs, evaluated with micro recall and F1.This weak-labeling evaluation targets scalable high-salience supervision rather than exhaustive word-level prosody annotation.
  • Annotation Capability: Gemini-3-pro has the highest recall and F1 for both stress and pause, but pause F1 remains moderate at 0.532.The labels are therefore used as scalable weak supervision, with final outputs evaluated by human listeners.

6 Experiments

Experiments evaluate LoopTTS for recovering Stage 2 flagged utterances, testing pipeline quality, Judge replacement, instruction following, and Refiner design choices. Human evaluations show targeted refinement improves recovery and prosodic control, while revealing trade-offs in similarity and transcription accuracy.

  • 6.2 LoopTTS Pipeline Evaluation: Human evaluation samples 100 Stage 2 flagged utterances per Neutral and Emotional condition from CosyVoice2-generated speech.The selected failure subset is intended to distinguish recovery gains from already accepted utterances.
  • 6.2 LoopTTS Pipeline Evaluation: A single LOOPTTS refinement round improves MOS over raw flagged audio and the strongest budget-aligned re-generation baseline under Neutral and Emotional conditions.The comparison gives re-generation the same initial output and judge-triggered one-correction budget, but without target instructions.
  • 6.2 LoopTTS Pipeline Evaluation: Listeners prefer LOOPTTS over raw flagged audio (85.67%), CosyVoice2 re-generation (77.00%), and EmoVoice re-generation (72.3%).For the EmoVoice comparison, 19/30 samples receive at least 7/10 agreement and 10/30 receive at least 9/10 agreement.
  • 6.2 LoopTTS Pipeline Evaluation: Replacing Gemini with Qwen3-Omni Thinking preserves recovery gains, yielding MOS scores of 4.09/3.98 under Neutral/Emotional conditions and 79.67% preference over raw flagged audio.The same Refiner is used while the inference-time Judge handles both diagnosis and instruction generation.
  • 6.3 Instruction-Following Comparison: The Refiner achieves the highest MOS-I on Pitch, Speed, Stress, and Pause, but shows less vivid emotional delivery than EmoVoice.This result supports using the Refiner as a Stage 3 corrective model rather than a standalone emotional TTS generator.
  • 6.4 Ablation Studies: Ablation results suggest Position-Weighting and Structural Alignment help instruction following, but modest MOS gaps and some better WER scores limit mechanistic conclusions.Removing Structural Alignment causes the largest Avg. MOS-I drop; removing position weighting produces a smaller but consistent MOS/MOS-I decrease.
  • 6.4 Ablation Studies: Removing the initial utterance slightly affects MOS but reduces SIM and MOS-I, indicating that audio conditioning helps preserve identity and avoid unnecessary edits.More aggressive prosodic modification can also increase WER when stressed syllables or deliberate pauses affect transcription.

7 Conclusion

The paper presents LOOPTTS as an offline Filter–Judge–Refiner framework that combines AudioLLM-based prosodic diagnosis with guided expressive re-synthesis. Experiments show improved instruction following and human-rated recovery on flagged utterances versus raw audio and open-loop re-generation, while broader evaluation and deployment remain future directions.

  • 7 Conclusion: LOOPTTS integrates prosodic diagnosis with guided expressive re-synthesis in an offline Filter–Judge–Refiner framework.The approach uses AudioLLMs for diagnostic weak supervision and a Refiner trained with contrastive prosody data and a position-aware objective.
  • 7 Conclusion: Bounded refinement improves human-rated recovery quality on Stage 2 flagged utterances compared with raw audio and practical open-loop re-generation baselines.The Refiner also improves fine-grained instruction following under the reported evaluation protocol.
  • 7 Conclusion: LOOPTTS permits inference-time Judge replacement, as shown by LOOPTTS-Q, although its training labels remain Gemini-derived.The conclusion identifies full-distribution human evaluation, stronger deployment baselines, tonal languages, and lower-cost open judges as future directions.

Limitations

The evaluation and design of LoopTTS are bounded by selective human testing, limited speech coverage, binary prosodic controls, and possible speaker-similarity changes during expressive re-synthesis.

  • Evaluation scope: Human pipeline evaluation covers Stage 2 flagged utterances rather than every initial-generation utterance.Appendix A provides full-pipeline accounting and objective metrics over 3,000 utterances instead.
  • Data and domain scope: Experiments focus on English read and emotional speech with limited speakers and corpora.Broader multilingual, multi-speaker, and speaking-style coverage remains future work.
  • Control granularity: Stress and pause annotations and controls are binary, excluding stress intensity and calibrated pause duration.Finer-grained labels are proposed for continuous or multi-level expressive control.
  • Re-synthesis trade-offs: Guided expressive re-synthesis rather than waveform-level local editing may change unrequested regions and speaker embeddings.The paper identifies speaker-similarity degradation as a trade-off of expressive or prosodic changes.

Ethics Statement

The ethics statement describes compensated human participation, public research datasets without private or identifiable data, planned academic release, and responsible-use concerns for controllable speech synthesis.

  • Human annotation and compensation: The study includes paid professional evaluators, diagnostic raters, and prosodic annotators compensated under local regulations and institutional guidelines.The participants included ten blind-listening evaluators, six diagnostic alignment raters, and professional prosodic annotators.
  • Data privacy and consent: REFINER-DB uses publicly available research speech corpora and excludes private, user-uploaded, and personally identifiable data.The contrastive instruction data are generated from those public datasets through the AudioLLM annotation pipeline.
  • Licensing and release: The authors plan to release research materials for non-commercial academic use subject to license and redistribution terms.The release includes code, construction scripts, prompts, checkpoints, annotations, and evaluation sample lists where license-compatible.
  • Responsible use: Controllable prosodic synthesis may be misused for deceptive or manipulative audio, motivating responsible use with human oversight.

A LoopTTS on scaled evaluation set

On a 3,000-utterance emotional synthesis set, LoopTTS is compared with direct re-generation and AudioLLM-reranked best-of-5 selection using WER, SIM, and UTMOS, alongside stage-level accounting.

  • Evaluation setup: The scaled evaluation uses 3,000 emotional synthesis utterances and reports WER, SIM, and UTMOS.WER measures recognition errors, SIM compares against a neutral speaker reference, and UTMOS estimates perceptual quality.
  • Baselines: Untargeted CosyVoice2 and EmoVoice re-generation are applied to the same 3,000 requests, alongside a best-of-5 CosyVoice2 alternative.Best-of-5 retains the candidate receiving the highest AudioLLM scores for target-emotion fidelity and overall quality.
  • Results: LoopTTS obtains the lowest WER and highest UTMOS while retaining comparable SIM.Judge-reranked best-of-5 remains below LoopTTS in WER and UTMOS and requires five generations plus five scoring calls per request.
  • Efficiency: LoopTTS refinement requires one AudioLLM instruction call and one Refiner generation for each flagged request.This contrasts with the five generations and five AudioLLM scoring calls required by best-of-5 selection.
  • Pipeline accounting: Stage-level accounting separates initial passes, Stage 1 recovery, Stage 2 flags, and the remaining flagged subset after one refinement round.

B Data Statistics

REFINER-DB contains approximately 42K contrastive tuples from six public corpora, with separate validation and test splits and paired neutral–emotional data designed to emphasize prosodic contrast.

  • Dataset composition: REFINER-DB contains approximately 42K training tuples sourced from six publicly available speech corpora.The sources include emotionally expressive corpora and LibriTTS read speech with natural prosodic variation.
  • Splits: The dataset includes non-overlapping validation and test splits of 1,796 and 500 tuples, respectively.There is no overlap in target recordings or text utterance instances across training, validation, and test sets.
  • Pair construction: Neutral–emotional pairs share text and speaker prompts, making prosody the dominant contrast without removing all acoustic differences.Neutral counterparts are retained only when their synthesis satisfies WER=0 and UTMOS>3.
  • Test composition: The 500 test tuples are divided into 200 annotation-capability, 100 diagnostic-capability, and 200 instruction-following examples.These subsets are non-overlapping and support distinct evaluations.

C Pipeline Evaluation Setup

The evaluation tests LoopTTS under controlled low-quality utterances, open-loop regeneration baselines, ablations, counterfactual instructions, repeated refinement, speaker-similarity measures, and prompt-only TTS comparisons. Results examine recovery quality, instruction robustness, preservation, computational cost, and practical operating limits.

  • Evaluation protocol: 2,000 utterances per condition are synthesized for Neutral and Emotional settings, with failed samples excluded after up to two additional regeneration attempts.Stage 1 retains utterances only when Whisper-large-v3-turbo yields WER≤3.0% against the target text and UTMOS>3.0.
  • Evaluation protocol: The comparison includes flagged audio, CosyVoice2 and EmoVoice open-loop regeneration, global-only refinement, one- and two-round LoopTTS, Qwen3-Omni judging, and human instructions.The open-loop CosyVoice2 baseline uses a different random seed with the same text, speaker prompt, and emotion instruction.
  • Evaluation protocol: Ten professional evaluators score matched samples blindly on MOS, while SIM uses the neutral speaker reference and paired preferences use 30 flagged samples.Conditions are shuffled and system identities are hidden; evaluators score each sample from 1–5.
  • Ablations: Removing the initial utterance changes MOS from 4.21 to 4.17, SIM from 0.68 to 0.62, and Avg. MOS-I from 4.21 to 3.94.The initial audio acts as an acoustic reference for speaker identity and unmodified prosodic attributes, supporting targeted correction.
  • Robustness: Counterfactual stress and pause instructions test wrong-position execution and correct-position preservation; WER still improves over pre-refinement audio.The small test does not fully measure false-edit rates on natural unrequested positions, and non-zero preservation does not eliminate under-diagnosis risk.
  • Iterative refinement: Objective gains saturate after one to two rounds, while SIM gradually decreases in later rounds, establishing one or two rounds as the practical operating point.Later rounds refine previously generated audio, and AudioLLM over- or under-diagnosis may contribute to unnecessary edits or residual defects.
  • Speaker similarity: The Refiner achieves the highest SIM-EXP while retaining neutral-reference SIM comparable to EmoVoice; CosyVoice2 has the highest neutral-reference SIM but the lowest SIM-EXP.CosyVoice2’s weaker target-emotion realization is reflected by Emotion MOS-I: 2.25, so lower neutral-reference SIM can reflect intended emotional variation rather than speaker drift.
  • Practical considerations: Refiner-DB construction is estimated at about $907.2 for 42K tuples, and prompt-only Qwen3-TTS and MOSS-TTS trials did not reliably execute fine-grained word-level edits.The prompt-only comparison is preliminary rather than a full MOS-style baseline; LOOPTTS instead uses a separate post-generation Refiner.
Loading 2608.28970v1…