Source-linked AI summary
WavAlign: Enhancing Intelligence and Expressiveness in Spoken Dialogue Models via Adaptive Hybrid Post-Training
Yifu Chen, Shengpeng Ji, Qian Chen, Tianle Liang, Yangzhuo Li, Ziqing Wang, Wen Wang, Jingyu Lu, Haoxiao Wang, Xueyi Pu, Fan Zhuo, Zhou Zhao
TL;DR
End-to-end spoken dialogue models still struggle to improve intelligence and expressiveness together, and direct preference optimization is unreliable for mixed text–speech outputs. WavAlign separates modality roles by applying preference updates to semantic tokens, anchoring speech with SFT, and adapting their mixture using rollout reliability. Across architectures and benchmarks, the method consistently improves semantic quality and speech expressiveness, while the study remains limited by sequence-level rewards and imperfect audio-judge calibration.
Problem
Current open-source spoken dialogue models often fail to achieve strong semantic quality and expressiveness simultaneously.
Method
WavAlign applies preference optimization to text tokens, anchors speech tokens with SFT, and dynamically gates updates using rollout reliability.
Results
Experiments across architectures and benchmarks show consistent gains in semantic quality and acoustic expressiveness.
Takeaways & Limitations
Separating semantic preference refinement from acoustic stabilization provides a single adaptive post-training framework for improving both intelligence and expressiveness.
Takeaways & Limitations
The study is limited to sequence-level reward signals, lacks PPO-based speech-token experiments, and uses audio judges with weaker reliability and calibration than semantic judges.
Abstract
from arXiv · showhide
End-to-end spoken dialogue models have garnered significant attention because they offer a higher potential ceiling in expressiveness and perceptual ability than cascaded systems. However, the intelligence and expressiveness of current open-source spoken dialogue models often remain below expectations. Motivated by the success of online reinforcement learning(RL) in other domains, one might attempt to directly apply preference optimization to spoken dialogue models, yet this transfer is non-trivial. We analyze these obstacles from the perspectives of reward modeling and rollout sampling, focusing on how sparse preference supervision interacts with dense speech generation under shared-parameter updates. Based on the analysis, we propose a modality-aware adaptive post-training recipe that makes RL practical for spoken dialogue: it constrains preference updates to the semantic channel and improves acoustic behavior via explicit anchoring, while dynamically regulating their mixture from rollout statistics to avoid unreliable preference gradients. We evaluate the method across multiple spoken dialogue benchmarks and representative architectures, and observe consistent improvements in semantic quality and speech expressiveness.
1 Introduction
End-to-end spoken dialogue models aim to combine semantic reasoning with expressive speech, but current systems struggle to improve both simultaneously. The paper diagnoses failures in unified preference optimization and proposes an adaptive hybrid strategy.
- Current open-source end-to-end systems leave substantial room to improve semantic quality, naturalness, and expressiveness simultaneously.
- Sparse acoustic rewards distributed across long speech-token sequences make credit assignment ill-conditioned and can enable reward hacking.
- Unified preference optimization can produce weak cross-modal coupling, gradient-energy imbalance, and noisy acoustic rewards.
- The proposed hybrid scheme applies preference optimization to text tokens, anchors speech tokens with SFT, and gates updates using rollout reliability.
- Experiments across architectures and benchmarks show consistent gains in semantic quality and acoustic expressiveness.
2 Related Works
Prior work often separates semantic intelligence from speech expressiveness because full mixed-modality optimization can destabilize cross-modal behavior. The related literature therefore motivates decoupled and modular approaches, while leaving unified optimization unresolved.
- Prior spoken-dialogue work typically targets either semantic quality or expressiveness and naturalness, leaving joint optimization elusive.
- Optimizing full mixed text–audio sequences can cause cross-modal instability and text–speech misalignment, motivating text-only or decoupled designs.
- EQ-oriented methods rely on reward modeling and preference data but can be brittle to reward hacking through spurious acoustic cues.
- Overall, fragmented objectives and persistent cross-modal instability leave a unified solution missing.
3 Methodology
The methodology models spoken replies as text and speech token sequences, then analyzes how SFT, preference optimization, and rollout properties interact across modalities. It uses text-restricted preference updates, speech anchoring, and adaptive hybrid weighting to stabilize semantic refinement and acoustic behavior.
- 3.1 Spoken Dialogue Model: Spoken dialogue models generate text and speech conditioned on an input context, with interleaved, parallel, and thinker–talker architectures represented uniformly as two token sequences.The representation uses text tokens yT and speech tokens yS while allowing architecture-specific conditioning contexts.
- 3.1 Spoken Dialogue Model: The joint conditional log-likelihood is partitioned by token type, enabling objectives to target text and speech positions separately.Token-subset restricted likelihood explicitly controls which token types receive preference gradients.
- 3.2 Policy Optimization: SFT supplies dense token-level supervision, whereas GRPO obtains online rollout rewards but shares sequence-level advantages across token positions, making long dense streams difficult to credit.A KL term provides a dense trust-region constraint that limits policy drift during GRPO optimization.
- 3.3 Observations: A unified mixed text–speech preference objective suffers from cross-modal trade-offs, weak gradient coupling, and reward dilution across dense speech tokens.Semantic updates have higher effective energy, while acoustic updates can be high-variance and destabilize prosody and timbre.
- 3.3 Observations: SFT produces larger coherent distribution shifts, while stabilized PO/RL remains local; reward judgments are also more informative for semantics than acoustics.These observations motivate using SFT for reliable global shifts and acoustic anchoring, while restricting preference-driven updates to text tokens.
- 3.4 Dynamic Hybrid Post-Training Objective: The proposed single-stage adaptive hybrid loop samples spoken replies, evaluates decoded audio with rewards, applies preference optimization to text tokens, and dynamically gates its mixture with SFT.The gate increases preference weight when rollouts contain an acceptable sample and show sufficient reward discrimination; at least 0.2 SFT remains as an acoustic safety anchor.
4 Experiments
Experiments evaluate intelligence and expressiveness across benchmarks, architectures, optimization scopes, and weighting strategies. The adaptive hybrid achieves strong IQ–EQ performance and is preferred by human raters.
- Experimental setup: Evaluation spans three benchmarks with 18 sub-tasks covering semantic competence and expressive capability across two end-to-end speech backbones.VoiceBench and OpenAudioBench assess semantic performance, while VStyle assesses paralinguistic control.
- Intelligence (IQ): Teacher-forced SFT consistently underperforms the base model on reasoning-heavy subsets, while full-token preference optimization is suboptimal across both backbones.The authors attribute the SFT result mainly to gradient interference and overwriting of pretrained reasoning behaviors.
- Intelligence (IQ): Restricting preference updates to text tokens yields more reliable IQ gains, and the dynamic hybrid achieves the strongest overall IQ among compared methods.The hybrid is described as mitigating catastrophic forgetting while preserving preference-learning benefits.
- Expressiveness (EQ): On VStyle, SFT remains highly competitive for style following and acoustic attributes, whereas full-token DPO causes severe expressive-speech degradation.These results are consistent with dense supervision imprinting paralinguistic realizations and acoustic preference signals being noisy or weakly discriminative.
- Expressiveness (EQ): The proposed method achieves the best aggregate EQ on both architectures while remaining close to the best style-following score.This produces a better IQ–EQ Pareto trade-off than either SFT or preference optimization alone.
- Weighting schemes and optimization scope: With the same fixed 0.5/0.5 mixture, text-token scope reaches 52.60/2.60 versus 48.70/2.48 for all-token scope in IQ/EQ, while dynamic gating with EMA reaches 55.24/2.92.Without EMA, the corresponding result is 53.15/2.53; a fixed SFT-favoring mixture improves EQ to 2.72 but lowers IQ to 49.94.
- Human evaluation: In a blinded study of 80 items with three raters per item, the proposed model significantly outperforms the baseline with an approximately 4:1 overall win-to-loss ratio.Annotators rated helpfulness and naturalness, and the model was preferred on both dimensions.
5 Conclusion
The paper analyzes why preference optimization is unreliable for mixed text–speech outputs and proposes adaptive hybrid post-training. The method stabilizes speech while improving intelligence and expressiveness across architectures and benchmarks.
- The analysis identifies semantic-signal dilution and acoustic drift as consequences of preference updates in end-to-end spoken dialogue models.
- The proposed adaptive hybrid method stabilizes speech while improving both intelligence and expressiveness across architectures and benchmarks.
Limitations.
The study is limited to sequence-level reward signals and does not evaluate PPO-based speech-token training. It also notes that audio judges are less reliable and calibrated than semantic judges.
- Limitations: The study focuses on sequence-level reward signals and does not run PPO-based speech-token experiments.The authors attribute the omission to limited resources.
- Limitations: Audio judges are not yet as reliable or well calibrated as text/semantic judges, so improved judges could change the reported findings.
A.1 Evaluation data and repeated-sampling protocol
The evaluation uses repeated stochastic sampling from spoken-dialogue checkpoints, with separate transcript-only semantic and audio-only acoustic ratings. Judge quality is assessed through global agreement and within-prompt ranking agreement, with semantic ranking proving more stable than acoustic ranking.
- Evaluation data and repeated-sampling protocol: The study covers vitaaudio and kimiaudio, each containing 40 prompt IDs split between audio-type and text-type prompts.Human-rated sample counts are 318 for vitaaudio and 319 for kimiaudio.
- Evaluation data and repeated-sampling protocol: The evaluation samples eight spoken responses per prompt ID from the same model under fixed decoding settings.Nucleus sampling uses temperature T=0.9 and top-p=0.9, with a maximum sequence length of 2048 tokens; some IDs have seven usable samples after failures.
- Evaluation data and repeated-sampling protocol: Human raters score semantic quality from transcripts and acoustic/paralinguistic quality from audio on separate 1–5 Likert axes.The acoustic rubric evaluates clarity, fluency, pronunciation/accent, prosody/pacing, and emotional appropriateness.
- Evaluation data and repeated-sampling protocol: Agreement analysis reports Pearson correlation, MAE, the ≤1 pass rate, and bias across all samples, plus Intra-ID Spearman for repeated-sample rankings.Multiple ratings are averaged per sample before global and within-prompt agreement metrics are computed.
- Evaluation data and repeated-sampling protocol: Across datasets and judges, semantic ranking agreement is consistently stronger than acoustic ranking agreement.This supports using semantic judgments as the primary within-prompt discriminator while maintaining speech feasibility through dense supervision.
B Model Output Diversity Experiments
The diversity experiments quantify within-prompt dispersion across repeated spoken outputs and connect those statistics to judge consistency and preference construction. Acoustic variation and discriminability are generally weaker than semantic variation and discriminability, motivating modality-separated training analysis.
- Model Output Diversity Experiments: Output diversity is measured as within-prompt dispersion across repeated spoken answers sampled for each prompt ID.The same repeated-sampling pool supports diversity analysis, judge agreement, and preference-pair construction.
- Model Output Diversity Experiments: Human-rated mean variance is 1.066 semantic versus 0.387 acoustic for vitaaudio, and 0.911 semantic versus 0.509 acoustic for kimiaudio.
- Model Output Diversity Experiments: Judge-rating plots place most semantic-variance versus acoustic-variance points below the diagonal, indicating weaker acoustic discriminability under repeated sampling.
- Model Output Diversity Experiments: The gradient analysis separates text-token and speech-token losses by computing modality-specific gradients through independent backward passes.This produces clean modality-separated gradients for layer-wise and global comparisons.
- Model Output Diversity Experiments: The analysis compares gradient statistics from GRPO, SFT, and DPO using empirical distributions, Mann–Whitney U tests, multiple-comparison correction, and Cliff’s δ.
- Model Output Diversity Experiments: Teacher-forcing analysis compares base and fine-tuned log probabilities across text-token and speech-token regions to examine how dense updates reshape speech-token likelihoods.
E.3 Self-built dataset construction (control/understanding)
The self-built datasets target style control and style understanding, while preference pairs are constructed from repeated candidate responses using semantic and acoustic judge scores. A margin-filtered weighted utility produces one DPO pair per prompt.
- E.3 Self-built dataset construction (control/understanding): Self-built datasets target style control and style understanding under explicit JSON schemas.The converted training pool contains 13,510 input samples.
- E.3 Self-built dataset construction (control/understanding): For each prompt, the process samples eight candidate spoken responses and assigns separate 1–5 semantic and acoustic scores.
- E.3 Self-built dataset construction (control/understanding): A fixed weighted sum converts semantic and acoustic scores into a single utility used to select preferred and rejected responses.Pairs are retained only when the utility gap exceeds a margin, with δ = 0.5 by default and semantic score used first for ties.
- E.3 Self-built dataset construction (control/understanding): The construction forms one DPO pair per prompt while preserving diversity from repeated sampling.The reward and judge prompts used for scoring are documented separately.
G Human Subjective Evaluation Protocol
The study uses a blinded side-by-side comparison of the proposed model and the Original Model, with randomized presentation and independent annotator judgments across content, delivery, and overall preference. Item-level majority outcomes are summarized as Win/Tie/Loss rates and tested with paired sign tests.
- Evaluation setup: The evaluation compares the proposed model against the Original Model using two spoken responses per item under identical input conditions.Model identities are hidden, and A/B ordering is randomized for each item and annotator.
- Evaluation setup: The evaluation set contains 40 items: 20 sampled from VOICEBENCH and 20 from VSTYLE.Prompts are kept unchanged and sampled without replacement unless otherwise specified.
- Evaluation setup: Each item is rated by 3 independent annotators, who may replay responses and are encouraged to use headphones in a quiet environment.
- Criteria and decision rule: Annotators judge Helpfulness, Naturalness, and Overall preference, selecting whether A is better, B is better, or the responses are tied.Helpfulness emphasizes instruction adherence and logical coherence; Naturalness covers prosody, timbre, fluency, and emotional appropriateness.
- Statistical testing: Majority votes determine each item’s outcome, after which Win/Tie/Loss rates are computed and two-sided paired preference sign tests are applied after excluding ties.The sign test models wins as Binomial(N, 0.5) under the null hypothesis of no preference.
H.1 Sensitivity Analysis of EMA Coefficient α
The sensitivity analysis examines how EMA smoothing affects the adaptive mixture of SFT and preference optimization. Moderate smoothing and group size provide a balance between training stability, semantic refinement, and computational cost.
- EMA coefficient α: Under-smoothing with α = 0.5 produces high variance in λt, destabilizing training updates.
- EMA coefficient α: Over-smoothing with α = 0.99 causes excessive lag, so λt rises too slowly when rollout quality improves.This leaves the model dominated by SFT and misses opportunities for semantic refinement through preference optimization.
- EMA coefficient α: The default α = 0.9 provides the best balance in the reported sensitivity analysis.
- Group size and EMA: Increasing group size to G = 8 yields further IQ gains but little improvement in EQ.
- Group size and EMA: Removing EMA causes a substantial performance drop regardless of group size, while G = 4 is selected for its trade-off between computational cost and overall performance.
- Dynamic weight trajectory: During training, λt rises as rollout quality and discriminability improve, then stabilizes between 0.35 and 0.55.The converged range indicates continued contributions from both SFT and preference optimization.