PaperlayerView exact revision on arXiv

Source-linked AI summary

Noise-Aware Lombard Speech Synthesis via Flow-GRPO for Intelligibility Optimization

Zhaoyang Wang, Dingding Yao, Chengzhong Wang, Junfeng Li

arXiv:2610.20181v1cs.SD

TL;DR

Conversational agents often remain insensitive to ambient noise, while existing Lombard-TTS approaches do not directly optimize intelligibility under changing conditions. NIL-TTS adds FiLM-based environmental conditioning to flow-matching TTS and follows supervised fine-tuning with Flow-GRPO post-training. The paper reports improved intelligibility in adverse conditions and Lombard-like acoustic correlates, alongside a small speaker-similarity decrease in noise-free evaluation.

  • Problem

    Conversational agents often lack acoustic adaptation, while existing Lombard-TTS approaches face data constraints and commonly optimize acoustic mimicry rather than intelligibility or environment-dependent dynamics.

    Source: §I · p. 1

  • Method

    NIL-TTS adds FiLM-based environmental conditioning to a flow-matching TTS model, then applies supervised fine-tuning followed by Flow-GRPO post-training.

    Source: §I · p. 1, 1) Base Flow-Matching Acoustic Model · p. 3, 2) Environment-Aware Injection via FiLM · p. 3

  • Results

    NIL-TTS significantly enhances speech intelligibility in adverse conditions and recovers key acoustic correlates of the human Lombard effect.

    Source: B. Acoustic Parameter Analysis · p. 9, §VI · p. 11

  • Takeaways & Limitations

    The authors present NIL-TTS as a practical framework for noise-aware speech synthesis in challenging real-world environments.

    Source: §VI · p. 11

    In noise-free evaluation, speaker similarity (SIM) decreases marginally from 0.76 to 0.75, despite lower WER and higher UTMOS than the F5-TTS baseline.

    Source: NOISE-FREE CONDITIONS · p. 12

Abstract

from arXiv · show

Voice-based conversational agents are increasingly deployed in real-world settings, where environmental noise often degrades the interactive experience. Therefore, robust acoustic adaptability is essential to maintain natural and intelligible human-computer communication. However, most existing systems remain acoustically insensitive and fail to emulate the human Lombard effect, in which speakers involuntarily adjust their vocal production to preserve intelligibility in the presence of noise. As a result, their communicative reliability degrades under adverse acoustic conditions. To remain intelligible in noisy environments, conversational agents require a mechanism to emulate noise-induced adaptive behavior during synthesis. We propose NIL-TTS, a noise-induced Lombard TTS framework. We integrate a feature-wise linear modulation (FiLM) mechanism into a flow-matching backbone for dynamic vocal adaptation. A two-stage training paradigm is utilized: initial supervised fine-tuning on paired Lombard corpora, followed by Flow-GRPO post-training to optimize for intelligibility. Evaluations confirm that NIL-TTS autonomously adjusts its output in response to varying noise conditions, exhibiting key correlates of the human Lombard effect. Driven by these adaptive modulations, the synthesized speech achieves significant intelligibility improvements that scale proportionally with noise intensity. Ultimately, this work presents a flow-matching and reinforcement learning-based advancement within the adaptive Lombard speech synthesis paradigm for challenging real-world scenarios.

I. INTRODUCTION

Conversational agents need to adapt to ambient acoustics, but existing systems remain acoustically insensitive and many Lombard-TTS approaches prioritize acoustic mimicry over intelligibility. NIL-TTS introduces an RL-driven framework for noise-aware Lombard synthesis, while F5-TTS provides its flow-matching foundation.

I. INTRODUCTION

  • Existing Lombard-TTS approaches often rely on limited corpora, SNR-based loudness adjustment, or style embeddings, while optimizing acoustic mimicry rather than intelligibility or environment-dependent dynamics.The paper identifies scarce high-quality Lombard data as a source of rigid mappings and poor generalization to unseen speakers.Source: §I · p. 1
  • The paper links reinforcement learning’s environmental interaction to the human auditory-vocal feedback loop, motivating RL for Lombard speech synthesis.It states that RL’s potential for Lombard synthesis had remained unexploited.Source: §I · p. 1
  • NIL-TTS is introduced as an RL-driven framework for noise-aware, high-intelligibility speech synthesis, addressing the lack of environment-driven adaptation and direct intelligibility optimization in existing approaches.The paper also reports acoustic adjustments under adverse conditions and intelligibility improvements from those shifts.Source: §I · pp. 1–2

A. F5-TTS

  • F5-TTS is a non-autoregressive flow-matching TTS system with a Diffusion Transformer backbone, using ConvNeXt V2 blocks and filler-token padding rather than explicit phoneme-level alignment.Its architecture is presented as a flexible foundation for environment-aware Lombard adaptation.Source: A. F5-TTS · p. 2

B. FiLM

NIL-TTS conditions flow-matching speech generation on ambient acoustics through FiLM and trains it with supervised fine-tuning followed by Flow-GRPO. GRPO uses group-relative rewards, and Flow-GRPO applies this approach directly to continuous mel-spectrogram generation.

B. FiLM

  • FiLM injects external acoustic conditions through feature-wise scaling and shifting, providing a pathway for environment-dependent vocal adaptation in the Diffusion Transformer.The paper describes FiLM as a bridge between ambient-noise perception and adjustments to the model’s generative trajectory.Source: B. FiLM · p. 2

C. Group Relative Policy Optimization (GRPO)

  • GRPO compares each output’s reward with its group’s mean and standard deviation, avoiding a separate critic model and reducing memory and computation costs relative to PPO.A KL penalty may constrain policy updates relative to a reference model.Source: C. Group Relative Policy Optimization… · p. 2

D. Flow-GRPO

  • Flow-GRPO applies online reinforcement learning directly in continuous mel-spectrogram space, using group-normalized rewards to refine flow trajectories without a critic.The approach optimizes the flow vector field against human-centered quality metrics.Source: D. Flow-GRPO · p. 2

III. METHODOLOGY

  • NIL-TTS combines environmental FiLM conditioning with supervised fine-tuning and Flow-GRPO post-training to learn Lombard acoustic priors and optimize vocal strategies for intelligibility.The pipeline overview distinguishes paired plain-and-Lombard speech SFT, SDE-based exploratory rollouts, and ODE-based deterministic inference.Source: §III · p. 2, Figure 1 · p. 3

A. Architectural Framework

NIL-TTS builds on a conditional flow-matching acoustic model and adds FiLM conditioning so ambient noise can shape speech generation. The environmental condition combines noise and SNR information and modulates transformer hidden states.

1) Base Flow-Matching Acoustic Model:

  • The conditional flow-matching model generates mel-spectrograms from text by learning a vector field from prior noise to target speech.The conditioning includes linguistic features and speaker identity features from a reference audio prompt.Source: 1) Base Flow-Matching Acoustic Model · p. 3

2) Environment-Aware Injection via FiLM:

  • A pre-trained noise encoder and an SNR projection provide environmental features that are fused into a global condition for the DiT.Average pooling removes frame-level noise dynamics before this utterance-level context is injected into hidden states.Source: 2) Environment-Aware Injection via FiLM · p. 3
  • FiLM generates block-specific scaling and shifting parameters to modulate hidden states, and is applied only in deeper transformer blocks (i > 2).Restricting modulation to deeper blocks preserves foundational phonetic alignment learned from clean speech.Source: 2) Environment-Aware Injection via FiLM · p. 3
  • Flow-GRPO uses noisy ASR transcription for intelligibility reward and a speaker encoder to penalize identity distortion.The objective is described as a proxy for human auditory feedback.Source: Figure 2 · p. 4

B. Two-Stage Training Paradigm

Training first uses paired plain and Lombard speech to establish noise-to-acoustic mappings, then applies Flow-GRPO post-training to optimize intelligibility. The post-training stage uses stochastic flow trajectories and rewards for noisy-condition transcription and speaker similarity.

1) Supervised Fine-Tuning (SFT):

  • SFT trains on paired plain and Lombard speech from the same speaker, conditioning on text, padded plain speech, and ambient noise.The masked flow-matching loss establishes a baseline policy for mapping noise conditions to basic Lombard acoustic features.Source: 1) Supervised Fine-Tuning (SFT) · p. 4

2) Flow-GRPO Post-Training:

  • Flow-GRPO addresses SFT's lack of an explicit intelligibility objective by post-training the policy against reward signals.SFT primarily optimizes data likelihood rather than intelligibility.Source: 2) Flow-GRPO Post-Training · p. 4
  • Because deterministic ODE generation lacks the stochasticity needed for RL exploration, generation is reformulated as an SDE that preserves the ODE's marginal distributions.The diffusion coefficient controls stochasticity, and the discretized update uses the Euler-Maruyama method.Source: 2) Flow-GRPO Post-Training · p. 4
  • The intelligibility reward evaluates ASR transcription after generated speech is mixed with target noise at a specified SNR.ASR feedback encourages adaptation to ambient noise and acoustic strategies that maximize transcription accuracy.Source: 2) Flow-GRPO Post-Training · p. 4
  • A speaker-similarity reward penalizes identity distortion, while group-normalized advantages and a clipped objective with KL regularization update the policy.The KL penalty anchors the policy to the SFT reference model.Source: 2) Flow-GRPO Post-Training · p. 5

C. Inference

Inference generates a deterministic Lombard speech sample for specified text and noise conditions, using a speaker reference. A vocoder converts the predicted mel-spectrogram into the final waveform.

Inference

  • Inference uses a deterministic ODE solver with a fixed number of integration steps, such as 32 NFE, for stable and reproducible output.A pre-trained vocoder synthesizes the predicted mel-spectrogram into the acoustic waveform.Source: C. Inference · p. 5

IV. EXPERIMENTS

The experiments use separate datasets for paired-style SFT, Flow-GRPO post-training, and zero-shot evaluation. Training and evaluation include speech and environmental-noise corpora spanning multiple domains and conditions.

A. Datasets

1) Supervised Fine-Tuning (SFT) Dataset:

  • DEMAND noise recordings from six environment categories augment SFT signals to establish foundational noise-to-acoustic mappings.The categories are Domestic, Office, Public, Transportation, Nature, and Street.Source: 1) Supervised Fine-Tuning (SFT) Dataset · p. 5

2) Flow-GRPO Post-Training Dataset:

  • Flow-GRPO uses 945 hours of quality-filtered Mandarin WenetSpeech4TTS-Premium speech and approximately 181 hours of DNS Challenge noise across 150 audio classes.The speech subset spans audiobooks, interviews, and podcasts; the noise includes office, transportation, and social environments.Source: 2) Flow-GRPO Post-Training Dataset · p. 5

3) Evaluation Dataset:

  • Zero-shot evaluation uses 2,020 Mandarin DiDiSpeech and 1,088 English Common Voice utterances, perturbed with Babble, Factory, Pink, and White noise.The utterances come from the Seed-TTS objective test configuration, and noise profiles come from NoiseX-92.Source: 3) Evaluation Dataset · p. 5

B. Model and Training Configurations

NIL-TTS adds environment sensing to the F5-TTS architecture and trains it first on paired Lombard speech, then with Flow-GRPO. Evaluation tests its noise-adaptive synthesis across noise types and SNRs.

1) Model Configuration:

  • The F5-TTS-based architecture encodes ambient noise and SNR, then injects the resulting environmental condition into DiT hidden states through FiLM.The noise encoder processes 44.1 kHz waveforms; the TTS pipeline and vocoder operate at 24 kHz.Source: 1) Model Configuration · pp. 5–6

2) Training Details:

  • SFT freezes the pretrained DiT backbone and trains only environment-aware FiLM modules, followed by 4,800 steps of on-policy Flow-GRPO post-training.SFT trains for 680,000 steps with SNR sampled from −7.0 to 5.0 dB; post-training samples SNR from U(−10, 15) dB and uses ASR WER and speaker-similarity rewards.Source: 1) Model Configuration · p. 6, 2) Training Details · p. 6

C. Evaluation

  • Evaluation benchmarks NIL-TTS synthesized speech on the Seed-TTS test set under Babble, Factory, Pink, and White noise across six SNR levels: 6, 3, 0, -3, -6, and -9 dB.Evaluation uses a deterministic ODE solver with 32 NFE and CFG strength 2.0.Source: C. Evaluation · p. 7

V. RESULTS AND ANALYSIS

NIL-TTS improves objective intelligibility over signal-processing and neural Lombard baselines across Mandarin and English conditions. Its advantage grows under severe noise, and the study reports transfer gains on zero-shot English synthesis after Mandarin-only Flow-GRPO training.

1) Objective Intelligibility Evaluation:

  • NIL-TTS has its largest relative advantage as noise worsens, while Babble is the hardest noise condition and white noise causes the smallest WER increase.At high SNR, the improvement over SFT is more modest; all models degrade as SNR decreases.Source: 1) Objective Intelligibility Evaluation · p. 7
  • At −6 dB on Seed-TTS test-zh, NIL-TTS reduces WER to 0.224 from 0.289 for SFT-only and 0.378 for the vanilla baseline.NIL-TTS also reaches ESTOI 0.298, versus 0.279 for SFT-only and 0.246 for the baseline; RL-only improves over baseline but trails NIL-TTS.Source: 1) Objective Intelligibility Evaluation · p. 7
  • Flow-GRPO training conducted only on Mandarin data also yields consistent intelligibility gains for zero-shot English synthesis under adverse noise.The authors interpret this transfer as evidence that adaptation operates primarily in acoustic rather than lexical space.Source: 1) Objective Intelligibility Evaluation · p. 7
  • NIL-TTS consistently outperforms signal-processing and neural Lombard baselines across evaluated SNRs in both Mandarin and English.The comparison includes SS, SSDRC, MetaVoice-1B Lomb, and F5-TTS-Style; the caption specifies WER and ESTOI averaged over four noise types at 0, −3, and −6 dB.Source: 1) Objective Intelligibility Evaluation · p. 8, Table I · p. 8
  • Unlike uniform acoustic filtering or PCA-based style manipulation, NIL-TTS senses the environment to shift vocal energy toward bands less affected by ambient noise.The full model also outperforms RL-only and SFT-only configurations, supporting the authors’ conclusion that both training stages contribute to the reported gains.Source: 1) Objective Intelligibility Evaluation · p. 8

2) Subjective Intelligibility Evaluation:

In a blind Babble-noise listening test, NIL-TTS improves subjective recognition over the baseline across all tested SNRs. Its advantage over SFT-only is significant under more severe masking, and accuracy gains increase as SNR falls.

2) Subjective Intelligibility Evaluation:

  • NIL-TTS significantly improves subjective intelligibility over the vanilla baseline at all tested SNRs (p < 0.001).The blind test focused on Babble noise at 0, −3, and −6 dB.Source: 2) Subjective Intelligibility Evaluation · p. 8
  • Compared with SFT-only, NIL-TTS shows no significant difference at 0 dB (p = 0.368), but improves recognition at −3 dB (p = 0.012) and −6 dB (p = 0.027).Source: 2) Subjective Intelligibility Evaluation · p. 9
  • At −6 dB, NIL-TTS raises transcription accuracy over the baseline from 0.367 to 0.598, with gains increasing monotonically as SNR drops.The test used 360 Mandarin clips balanced across four model configurations and three SNRs, with twelve listeners transcribing clips.Source: 2) Subjective Intelligibility Evaluation · pp. 8–9

B. Acoustic Parameter Analysis

Flow-GRPO refines rather than uniformly amplifies Lombard-related acoustic cues, while NIL-TTS adds little inference-time overhead relative to vanilla F5-TTS.

B. Acoustic Parameter Analysis

  • As SNR decreases, both models generally raise F0, F1, and F2 and flatten spectral tilt, but NIL-TTS retains lower absolute values and a steeper tilt than SFT-only.The authors interpret this pattern as Flow-GRPO rebalancing acoustic adaptation rather than simply amplifying cues.Source: B. Acoustic Parameter Analysis · p. 9
  • Under lower SNR, NIL-TTS spectrograms show upward harmonic shifts and greater energy concentration in higher frequencies.The authors relate this redistribution to the within-model reduction in spectral tilt as noise becomes more severe.Source: B. Acoustic Parameter Analysis · p. 9, Figure 6 · p. 10

C. Computational Efficiency Analysis

  • NIL-TTS increases model size from 336M to 382M parameters while maintaining a competitive RTF of 0.27 versus 0.26 for vanilla F5-TTS.The reported RTF increase is 0.01; MetaVoice-1B Lomb has an RTF of 3.04, while F5-TTS-Style Lomb is reported at 387M parameters and RTF 0.27.Source: C. Computational Efficiency Analysis · p. 9

D. Ablation Studies on Post-Training Design Choices

The ablations examine Flow-GRPO design choices, while the conclusion reports that NIL-TTS improves intelligibility on Seed-TTS and recovers key human Lombard acoustic correlates.

D. Ablation Studies on Post-Training Design Choices

  • Because sparse rewards can destabilize reinforcement learning, the study ablates Flow-GRPO hyperparameters for their effects on final intelligibility.Table III reports WER across four noise types and SNRs of 0, −3, and −6 dB on the Seed-TTS test set.Source: D. Ablation Studies on Post-Training… · p. 10, Table III · p. 11
  • β=0 yields superior overall intelligibility; nonzero KL penalties can restrict policy updates, so stability instead relies on SFT initialization and speaker-similarity constraints.The authors attribute the tolerance for unconstrained optimization to stochastic SDE sampling around the learned probability path.Source: D. Ablation Studies on Post-Training… · p. 10
  • 32 rollout steps produce better WER metrics than 8 or 16, potentially because clearer acoustic cues require finer-grained sampling.The paper notes this differs from the original Flow-GRPO study, which found fewer rollout steps could improve convergence and metrics.Source: D. Ablation Studies on Post-Training… · pp. 10–11
  • The diffusion coefficient a=0.4 outperforms a=0.2 and a=0.6, balancing phonetic structure against the trajectory diversity needed to discover Lombard compensations.The authors report that excessive noise disrupts phonetic structure for the ASR reward, while insufficient noise limits exploration.Source: D. Ablation Studies on Post-Training… · p. 11

VI. CONCLUSION

  • NIL-TTS significantly enhances intelligibility in adverse conditions on Seed-TTS and recovers key acoustic correlates of the human Lombard effect.The framework combines FiLM-based environmental conditioning with a flow-matching backbone and uses supervised fine-tuning followed by Flow-GRPO post-training.Source: §VI · p. 11

APPENDIX A VALIDATION OF ASR-BASED REWARDS FOR INDUCING

The paper checks whether ASR rewards distinguish Lombard from plain speech under noise, describes the objective-evaluation scope, and reports noise-free comparisons with F5-TTS.

LOMBARD SPEECH UNDER ADVERSE CONDITIONS

  • The ASR-reward validation addresses whether recognition accuracy tracks human Lombard intelligibility, a concern because misaligned feedback could produce policies that do not improve perceived intelligibility.The study evaluates plain, moderate Lombard, and strong Lombard speech under environmental noise.Source: LOMBARD SPEECH UNDER ADVERSE CONDITIONS · p. 11
  • Across four ASR architectures and the tested noise conditions, stronger Lombard speech consistently has lower WER than plain speech, with the gap widening as SNR falls.The authors use SenseVoice-Small for post-training rewards, citing the trade-off between performance and inference speed.Source: LOMBARD SPEECH UNDER ADVERSE CONDITIONS · p. 11, Table IV · p. 12

APPENDIX B DETAILED OBJECTIVE INTELLIGIBILITY SCORES

  • The objective evaluation covers WER and ESTOI across six SNR levels, four noise types, and Seed-TTS test-zh and test-en, with paired-utterance bootstrap 95% confidence intervals.Table V includes vanilla, RL-only, SFT-only, and NIL-TTS configurations.Source: Appendix B · pp. 11–12

NOISE-FREE CONDITIONS

  • With FiLM disabled for plain-speech synthesis, NIL-TTS has lower WER and higher UTMOS than F5-TTS, while speaker similarity falls slightly from 0.76 to 0.75.The authors hypothesize that style changes toward clearer articulation may be perceived by the SIM metric as reduced speaker identity.Source: NOISE-FREE CONDITIONS · p. 12

Related papers

Loading 2610.20181v1…