Source-linked AI summary
AuraSE: Low-Hallucination Generative Speech Enhancement via Multimodal Flow Matching and Inference Policy Optimization
Yingda Shen, Yao Qian, Yuxuan Hu, Junan Zhang, Yuxiang Wang, Hardik Hansrajbhai Chauhan, Yudong Li, Yufei Xia, Yufei Liu, Zhizheng Wu
TL;DR
Speech enhancement must improve perceptual quality while limiting hallucinations that alter content or speaker identity. AuraSE combines transcript-conditioned multimodal flow matching with inference policy optimization, and AuraSE-IPO leads across synthetic metrics and real-recording evaluations.
Problem
Speech enhancement seeks perceptual quality while limiting hallucinations that alter content or speaker identity.
Source: Discussion · p. 7, Conclusion · p. 7Method
AuraSE pairs a double-stream MMDiT using transcripts as a content anchor with IPO, which turns sample-dependent decoder behavior into periodically refreshed relative supervision.
Source: Text-Speech Double-Stream MMDiT · p. 3, Conclusion · p. 7Results
AuraSE-IPO ranks first on 11 of 12 metrics across synthetic test sets and has the best DNSMOS and input-referenced blind-listening scores among evaluated systems on real recordings.
Source: Main Results on Synthetic Test Sets · p. 6, Conclusion · p. 7Takeaways & Limitations
Deployment uses a fixed 10-step ODE decoder without classifier-free guidance or per-utterance configuration search.
Source: Main Results on Synthetic Test Sets · p. 6
Abstract
from arXiv · showhide
Generative speech enhancement models can produce cleaner and more natural-sounding speech than conventional discriminative approaches, but may hallucinate by changing speech content or speaker identity, even with transcript conditioning. We present AuraSE, a flow-matching framework that addresses hallucination through complementary modality and inference designs. First, a double-stream-to-single-stream multimodal Diffusion Transformer (MMDiT) allows transcript and acoustic representations to interact while preserving a dedicated pathway for the degraded input. Second, we find that the best decoder configuration, governed by guidance scale, sampling temperature, and step count, varies substantially across utterances. This observation motivates Inference Policy Optimization (IPO), an online, on-policy preference optimization method. IPO generates multiple candidates from the current model under different inference configurations, ranks them with a multi-objective reward, and learns from their relative preferences. AuraSE-IPO ranks first on 11 of 12 metrics across the synthetic test sets and obtains the highest DNSMOS and blind-listening scores among the evaluated systems on the real DNS blind test set. At deployment, it uses a fixed $10$-step ODE decoder without classifier-free guidance (CFG) or per-utterance configuration search.
Post-Training and Reinforcement Learning
Reinforcement-learning preference optimization has spread across generative modalities, while AuraSE combines multimodal flow matching with an inference-policy optimization stage.
Post-Training and Reinforcement Learning
- Reinforcement-learning post-training is now standard for aligning generative models with human preferences across text, vision, and speech.Source: Post-Training and Reinforcement Learning · p. 2
- Speech-restoration preference training has used PESQ adversarial optimization, NISQA-based PPO, and UTMOS-based DPO; IPO instead explores ODE inference configurations and refreshes preferences.Prior approaches include MetricGAN (Fu et al. 2019), PPO (Kumar, Perrault, and Williamson 2025), and DPO (Li et al. 2025b); closest methods use offline DPO or online GRPO.Source: Post-Training and Reinforcement Learning · p. 2
- AuraSE combines a multimodal flow-matching backbone with an inference policy optimization stage.The method takes a noisy waveform, extracts a degraded mel-spectrogram, and reconstructs the enhanced waveform with a neural vocoder.Source: Method · p. 2
Multimodal Flow Matching
AuraSE uses conditional flow matching to move from noise toward a clean mel-spectrogram while conditioning its learned velocity field on degraded audio and optional text.
Multimodal Flow Matching
- The conditional flow-matching path runs from noise at t = 0 to an approximately clean endpoint at t = 1, with residual noise controlled by σmin.Source: Multimodal Flow Matching · p. 2
- The enhancement model predicts a velocity field conditioned on the degraded mel-spectrogram and optional text tokens.Source: Multimodal Flow Matching · p. 2
- For variable-length utterances, the training loss is masked over valid mel frames.Source: Multimodal Flow Matching · p. 2
- At inference, the model integrates its learned velocity field from noise toward clean speech using the same cosine schedule over K steps.The CFG scale, initial-latent sampling temperature τ, and step count K define the inference policies studied.Source: Multimodal Flow Matching · p. 2
Text-Speech Double-Stream MMDiT
AuraSE’s double-stream MMDiT keeps mel and transcript representations separate through joint attention before merging them for acoustic flow prediction. IPO complements this backbone with on-policy candidate generation across inference policies and reward-ranked preference optimization.
Text-Speech Double-Stream MMDiT
- Double-stream blocks keep mel and text projections separate while jointly attending, then merge the streams into single-stream DiT blocks; only mel tokens predict flow velocity.Source: Text-Speech Double-Stream MMDiT · p. 3
- Unlike GRPO’s SDE trajectories, IPO uses ODE outputs across inference policies and requires no trajectory likelihood.Source: Reinforcement Learning Baselines: DPO… · p. 3
- IPO varies CFG scale, sampling temperature, and step count to generate same-input candidates, then samples preference pairs according to multi-objective reward gaps.Source: Inference Policy Optimization · p. 4
- IPO refreshes candidates and its reference from the updated model after bounded-reuse optimization, keeping preference data aligned with the evolving policy.Source: Inference Policy Optimization · p. 4, DPO GRPO IPO · p. 4
- Bounded buffer reuse amortizes rollout and reward costs, while longer reuse eventually makes preference pairs stale.Source: DPO GRPO IPO · p. 4
The Inference Landscape Is Sample-Dependent
Across eight inference configurations, the reward-best setting varies by utterance, and the resulting reward gaps offer preference data for IPO. Varying inference policy produces a stronger candidate signal than the compared alternatives.
The Inference Landscape Is Sample-Dependent
- Best-of-eight selection yields a better WER and quality trade-off than every tested fixed policy.The comparison covers eight policies spanning CFG scale, temperature, and step count.Source: The Inference Landscape Is… · p. 5
- The per-utterance winner spans all eight policies, with the most frequent chosen by only 18.3% of utterances.The most frequent winner compares with 12.5% under uniform selection.Source: The Inference Landscape Is… · p. 5
- Inference-policy variation yields the largest reward spread and highest mean reward among the three tested preference sources.The alternatives are reseeded default-policy ODE noise and a single-window SDE perturbation.Source: The Inference Landscape Is… · p. 5
Main Results on Synthetic Test Sets
On the synthetic benchmark, AuraSE-IPO ranks first on 11 of 12 metrics, while the base AuraSE maintains low WER under reverberation. Deployment uses a fixed 10-step policy without per-utterance search.
Main Results on Synthetic Test Sets
- AuraSE-IPO ranks first on 11 of 12 synthetic benchmark metrics; the exception is speaker similarity without reverberation.The benchmark includes 824 reverberant and 824 non-reverberant utterances.Source: Main Results on Synthetic Test Sets · p. 6, Table 3 · p. 6
- Under reverberation, SGMSE and StoRM exceed 20% WER, whereas AuraSE keeps WER low while matching or exceeding their perceptual quality.The synthetic benchmark reports both reverberant and non-reverberant conditions.Source: Main Results on Synthetic Test Sets · pp. 5–6, Table 3 · p. 6
- Deployment uses the fixed policy (γ, τ, K) = (0, 1.0, 10), with no per-utterance search.Transcript generation remains an external ASR cost.Source: Main Results on Synthetic Test Sets · p. 6
Real-World Evaluation
On real DNS blind-test recordings, AuraSE-IPO has the best DNSMOS scores among evaluated systems and the highest input-referenced listening score. The listening composite includes hallucination-sensitive dimensions alongside perceptual quality.
Real-World Evaluation
- AuraSE-IPO attains the best DNSMOS on all three axes among evaluated systems and the highest blind-listening score, 3.74±0.13.The human score is reported with a 95% confidence-interval half-width.Source: Real-World Evaluation · p. 6
- Fifteen experienced listeners rated random subsets of 200 enhanced clips against their noisy inputs, with at least five ratings per clip.The clips cover 40 inputs and five systems.Source: Real-World Evaluation · p. 6
- The input-referenced composite covers clarity, content fidelity, speaker similarity, hallucination, residual noise, and distortion.It evaluates perceptual quality and hallucination rather than generic audio quality alone.Source: Real-World Evaluation · p. 6
Architecture and Text-Conditioning Ablation
The ablation finds that AuraSE’s double-stream design uses transcript information more effectively than a single-stream design, and that content fidelity depends on transcript information rather than merely having a text stream.
Architecture and Text-Conditioning Ablation
- With ground-truth text, the double-stream design reaches 7.69% WER versus 10.97% for the single-stream design, while also improving SIM and SBS.The comparison is on the 300-utterance ablation subset.Source: Architecture and Text-Conditioning… · p. 6
- Removing text degrades WER to 12.51%, while ASR text retains most of the gain at 8.81%.The comparison indicates that the gain is driven by transcript content, not merely the presence of a text stream.Source: Architecture and Text-Conditioning… · p. 6
- The authors attribute the double-stream design’s text use to a separate acoustic pathway that lets text correct content while preserving speaker and signal detail.This interpretation is consistent with the reported SIM and SBS improvements.Source: Architecture and Text-Conditioning… · p. 6
Reinforcement Learning Results
IPO improves quality and fidelity together, outperforming DPO and GRPO in the reported RL comparison. Buffered updates produce smoother reward trajectories, while the paper scopes its claims to flow-matching speech enhancement.
Reinforcement Learning Results
- The best-of-eight base oracle reaches OVRL 3.271 and WER 5.87%, but is non-deployable; GT-DPO worsens WER and SIM.GT-DPO reduces OVRL while sharply degrading WER and SIM.Source: Reinforcement Learning Results · p. 7
- IPO improves OVRL and WER together, is best on every trained-method metric, and beats the best-of-eight base on OVRL, SIM, and SBS.It also outperforms GRPO overall, while offline IPOoff beats DPO and full online IPO beats online DPO.Source: Reinforcement Learning Results · p. 7
- An intermediate buffer has the smoothest EMA reward trajectory and highest reward; an over-large buffer settles lower, while buffer-free IPO beats GRPO and online DPO.Reward spread among rollout policies shrinks during training, though reduced output diversity may also contribute.Source: Figure 3 · p. 7, Reinforcement Learning Results · p. 7
- Improvements transfer to held-out synthetic conditions, unseen real recordings, and an independent input-referenced blind test; WER is scored against reference, not conditioning, transcripts.The ASR transcript is fixed across variants, and dependence on an external recognizer remains.Source: Discussion · p. 7
- Online RL remains costly because each rollout requires vocoding and metric evaluation, so the authors scope their claims to flow-matching speech enhancement.The conclusion reports a fixed ODE decoder at deployment without per-utterance configuration search.Source: Discussion · p. 7, Conclusion · p. 7