Source-linked AI summary
AlignAudio: A Unified Dual-Alignment Framework for Noise-Robust Speech-Conditioned Generation
Yixuan Li, Xuenan Xu, Zeyu Xie, Mengyue Wu
TL;DR
Environmental noise degrades speech-conditioned generation, which has largely been studied with clean speech. AlignAudio aligns conditioning representations and generation dynamics, and is reported to improve noisy-condition fidelity across STA and VC while retaining competitive clean-speech performance.
Problem
Speech-conditioned generation has largely been studied with clean speech, while environmental noise degrades performance and realistic interference remains insufficiently studied.
Source: §1 · p. 1Method
AlignAudio uses dual alignment to bring noisy conditioning representations toward clean ones and regularize conditional generation dynamics across clean and noisy settings.
Source: §1 · p. 2, §2.2 · p. 2, §2.3 · p. 3Results
AlignAudio improves noisy-condition conditioning fidelity on STA and VC, generalizes to challenging out-of-domain conditions, and maintains competitive clean-speech performance.
Source: §1 · p. 2Takeaways & Limitations
The framework supports single- and multi-stream conditioning and integrates with Flow Matching, MeanFlow, and Flow-GRPO post-training.
Source: §1 · p. 2, §4 · p. 4Future work will incorporate larger, more diverse real-world noisy datasets, including spontaneous speech.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Speech-conditioned generation maps speech signals to target audio outputs. Prior end-to-end speech-conditioned generative models achieve high fidelity in clean acoustic conditions, but their performance degrades substantially under environmental noise. While prior noise-robust approaches have achieved significant success in feature-level purification, maintaining consistent generation under clean and noisy conditioning remains a key challenge. To address this, we propose AlignAudio, a noise-robust generative framework based on dual alignment: (i) aligning the representations of clean and noisy speech to preserve semantic cues, and (ii) enforcing generation-level consistency within a flow matching paradigm to improve conditioning fidelity. Experiments on speech-to-audio (STA) and voice conversion (VC) tasks demonstrate that AlignAudio maintains performance comparable to baselines on clean speech while significantly outperforming them across various noisy conditions. Furthermore, the proposed dual-alignment strategy is applicable to both single- and multi-stream speech conditioning and can be integrated into different flow matching-based generation frameworks and post-training settings. Audio demonstrations and qualitative comparisons: \href{https://alignaudio.github.io/AlignAudio}{\textcolor{cyan}{\textit{https://alignaudio.github.io/AlignAudio}}}.
1. INTRODUCTION
Speech-conditioned generation supports tasks such as speech-to-audio and voice conversion, but real-world noise degrades performance and remains insufficiently studied. AlignAudio addresses this with dual alignment and reports stronger noisy-condition results across STA and VC while maintaining competitive clean-speech performance.
1. INTRODUCTION
- Speech-conditioned generation maps spoken input to diverse audio outputs, including environmental-sound synthesis in STA and speaker-identity transformation in VC.The cited research context includes diffusion models, flow matching, and language-modeling approaches.Source: §1 · p. 1
- Most prior speech-conditioned generation studies assume clean speech, although real-world noise degrades performance and leaves generation under realistic interference insufficiently studied.Preliminary STA experiments report approximately 33% higher FAD and around 18% lower CLAP under environmental noise.Source: §1 · p. 1
- Prior noise-robust methods mainly address a specific conditioning stream, while residual perturbations can still affect generation dynamics and accumulate into degraded quality and fidelity.The challenge is especially relevant when multiple heterogeneous inputs jointly influence the generation trajectory.Source: §1 · p. 1
- AlignAudio aligns clean and noisy conditioning representations, then regularizes generation dynamics to keep clean- and noisy-conditioned trajectories consistent.The framework uses lightweight adapters for representation alignment and constrains the conditional trajectory under clean and noisy settings.Source: §1 · p. 2, Figure 2 · p. 2
- AlignAudio improves conditioning fidelity on noisy STA and VC, generalizes to challenging out-of-domain conditions, and maintains competitive performance on clean speech.The framework is presented as robust across noisy conditions and as applicable to both single- and multi-stream conditioning.Source: §1 · p. 2
2. NOISE-ROBUST GENERATION VIA DUAL-ALIGNMENT
AlignAudio combines representation-level alignment with generation-level alignment to improve robustness by aligning conditioning representations and generation trajectories.
2. NOISE-ROBUST GENERATION VIA DUAL-ALIGNMENT
- AlignAudio pairs representation-level alignment with generation-level alignment to improve robustness across conditioning representations and generation trajectories.The framework is described for single- and multi-stream conditioning.Source: §2 · p. 2, §2.4 · p. 3
2.1. Preliminaries
- MeanFlow predicts an interval-averaged field using a Jacobian-Vector Product training target, enabling single- or few-step sampling.The average velocity over [r, t] is defined as displacement divided by interval length.Source: §2.1 · p. 2
2.2. Representation-level Alignment
- Lightweight adapters are trained by distillation to match noisy- and clean-input outputs to embeddings from a frozen, clean-trained conditioning network.The loss trades off noisy and clean adaptation, and adapter placement depends on the downstream architecture.Source: §2.2 · pp. 2–3
2.3. Generation-level Alignment
- Cosine directional alignment regularizes noisy-conditioned predictions against a fixed clean-flow reference, avoiding the potentially over-constraining magnitude agreement of pointwise L2 alignment.The clean and noisy predictions share state and time samples.Source: §2.3 · p. 3
- For standard flow matching, alignment is weighted more strongly at early timesteps and attenuated later; MeanFlow uses constant weighting.The weighting curve is controlled by an inflection point and slope, while MeanFlow disables temporal weighting.Source: §2.3 · p. 3
2.4. Task Instantiations
- The framework is instantiated for single-stream STA and dual-stream VC, with a further Flow-GRPO post-training adaptation for VC.STA uses SpeechT5 adapters and a flow-matching DiT; VC applies adapters to the multi-input fusion module and uses a MeanFlow-based DiT.Source: §2.4 · p. 3
3. EXPERIMENTS AND RESULTS
Experiments evaluate AlignAudio on STA and VC under synthetic, out-of-distribution, and recorded acoustic conditions. Ablations examine the contributions of both alignment stages and STA weighting schedules.
3.1. Experimental Setup
- The evaluation covers synthetic noise, DEMAND-based OOD noise, and recorded clean, room, street, cafe, and subway speech conditions.STA uses AudioCaps captions; VC training uses LibriTTS, with testclean speakers disjoint from training speakers. Ten listeners evaluated each task’s subjective metrics.Source: §3.1 · p. 3
3.2. Main Results
- Across noisy and recorded conditions, AlignAudio leads key STA and VC metrics; Flow-GRPO further yields the lowest WER across four conditions.STA CLAP is 0.412 for ST5 + FM in clean conditions, 0.339 under noise, 0.367 with front-end SE, and 0.421 with AlignAudio-STA. AlignAudio-VC* WER is 32.36% vs. Noro’s 45.27% under noise and 29.41% vs. 42.18% OOD; on recorded speech, AlignAudio-STA leads FAD and CLAP, full VC has lowest WER, and VC* has highest SECS.Source: Table 1 · p. 4, §3.2 · p. 4
3.3. Ablation Studies
- VC WER falls from 87.54% to 32.36% with generation-level alignment, while STA CLAP rises from 0.387 to 0.421.Removing representation-level alignment degrades both tasks; full dual alignment also outperforms noisy augmentation and joint training, supporting sequential adaptation of representations and generation dynamics.Source: §3.3 · p. 4, Table 2 · p. 4
- Early-t achieves the lowest STA FAD (1.88) among evaluated weighting schedules, while CLAP varies within 0.418–0.424.The paper interprets this as improved acoustic fidelity while preserving semantic alignment.Source: §3.3 · p. 4
4. CONCLUSION
AlignAudio aligns conditioning representations and generation dynamics to improve noise robustness in speech-conditioned generation. The framework supports STA and VC settings, with future work targeting broader noisy-data and application coverage.
4. CONCLUSION
- AlignAudio aligns conditioning representations and generation dynamics, supporting single- and multi-stream conditioning with flow matching and MeanFlow, plus Flow-GRPO post-training for VC.Experiments on STA and VC show improved conditioning fidelity under synthetic noise and real-world recorded interference; future work targets larger, more diverse real-world noisy datasets, spontaneous speech, and additional audio applications.Source: §4 · p. 4
5. COMPLIANCE WITH ETHICAL STANDARDS
The study used anonymous human-listener evaluation of audio samples and collected no personally identifiable information.
5. COMPLIANCE WITH ETHICAL STANDARDS
- Audio-sample listening evaluations were anonymous, collected no personally identifiable information, and required no institutional ethical approval.Source: §5 · p. 5