Source-linked AI summary
Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis
Sanyuan Chen, Min-Jae Hwang, Sho Inoue, Anna Sun, Bokai Yu, David Kant, Dongmin Hyun, Dorian Desblancs, Gregory Antonovsky, Oleg Repin, Peng-Jen Chen, Xutai Ma, Zehai Tu, Juan Pino, Wei-Ning Hsu
TL;DR
Text-AB addresses the challenge of high-quality cross-lingual voice dubbing and natural full-duplex dialogue synthesis. It combines alignment-free latent diffusion with large-scale pretraining and task-specific fine-tuning, achieving strong dubbing gains and improved dialogue human-likeness across short- and long-form settings.
Problem
Cross-lingual dubbing and full-duplex dialogue synthesis remain challenging because systems must preserve speaker characteristics across languages and produce controllable, fluid conversations.
Method
Text-AB uses a 3B-parameter flow-matching DiT with DAC-VAE latent audio, raw-text cross-attention, and fine-tuning for dubbing, dialogue, and emotional dialogue.
Results
Text-AB substantially improves internal-model dubbing scores and long-form dialogue human-likeness while approaching real recordings on short-form conversations.
Takeaways & Limitations
The results support a unified alignment-free latent-diffusion approach for scalable single-speaker, cross-lingual, and two-channel speech generation.
Abstract
from arXiv · showhide
We present Alignment-Free Text-Audiobox (Text-AB), a unified framework for high-quality voice dubbing and full-duplex dialogue synthesis. Building on a Diffusion Transformer trained with a flow-matching objective, Text-AB departs from the Audiobox system along three dimensions. First, it operates in a latent diffusion framework using DAC-VAE features that encode 48 kHz waveforms into a 25 Hz latent sequence, giving over 10x higher compression than previous EnCodec representations while improving resynthesis quality. Second, Text-AB is alignment-free: it consumes raw text via an off-the-shelf text encoder and learns text-speech alignment through cross-attention, removing the need for forced alignment and explicit duration prediction. Third, we scale model and data substantially, pretraining a 3B-parameter model on 480k hours of monolingual speech, followed by supervised fine-tuning on three downstream tasks: cross-lingual voice dubbing, full-duplex dialogue synthesis, and emotional full-duplex dialogue synthesis. At inference, Text-AB supports one-shot generation for up to ~1 min of speech and arbitrarily long-form generation via a multi-diffusion scheme, plus a multi-stage reranking strategy that enhances quality based on automated metrics. On a real-world dubbing benchmark, Text-AB delivers a step-change improvement over the latest internal dubbing system, with large gains in prosody similarity, voice similarity, naturalness, and shareability. For full-duplex dialogue synthesis, it approaches human recordings on short-form conversations and substantially outperforms the latest internal model on long-form human-likeness and expressivity, while natively modeling turn-taking, back-channeling, and emotional dynamics. For emotional dialogue synthesis, emotion conditioning significantly improves emotion alignment and emotional interaction quality over the unconditioned baseline.
1 Introduction
Text-AB targets challenging cross-lingual voice dubbing and full-duplex dialogue synthesis with an alignment-free, scalable framework. It substantially improves dubbing quality and long-form dialogue human-likeness while supporting natural conversational dynamics.
- Motivation: The framework addresses cross-lingual dubbing and full-duplex dialogue, where preserving speaker characteristics across languages and controlling conversational content remain challenging.Existing dialogue pipelines also limit fluidity by stitching single-turn monologues algorithmically.
- Scaling and training: The system scales to a 3B-parameter model pretrained on 480k hours, followed by task-specific fine-tuning for dubbing and dialogue synthesis.The evaluated downstream tasks include voice dubbing, full-duplex dialogue synthesis, and emotional dialogue synthesis.
- Voice dubbing: Text-AB substantially outperforms the latest internal dubbing model, gaining +0.39 shareability, +0.34 prosody similarity, +0.32 voice similarity, and +0.42 voice naturalness.These human-evaluation gains use a [−3, 3] MOS scale.
- Inference and dialogue modeling: Text-AB supports one-shot generation up to about one minute and arbitrarily long-form generation through multi-diffusion.The dialogue model directly learns turn dynamics, back-channeling, and emotional evolution rather than relying on heuristic stitching or insertion.
- Full-duplex dialogue synthesis: Text-AB nearly matches real recordings on short-form dialogue and improves long-form human-likeness over the previous system by +0.86 MOS.The short-form gap is −0.09 overall human-likeness on a 5-point MOS scale.
2 Text-AB
Text-AB combines flow matching, latent DAC-VAE audio representations, raw-text conditioning, and mono/stereo architectures. Its design masks inference–training mismatches while enabling single- and two-channel speech generation.
- 2.1 Flow-Matching: Flow matching trains the model to predict the velocity that moves a noised latent sample toward the target audio distribution.At inference, an ODE solver integrates the predicted velocity from Gaussian noise to generate the target latent.
- 2.1 Flow-Matching: Inference begins from Gaussian noise and solves an ODE with a fixed first-order Euler schedule to recover the generated latent.The number of time steps is tailored to the model.
- 2.2 Architecture: Text-AB-Mono and Text-AB-Stereo generate single- and two-channel speech, with stereo modeling jointly predicting both channels from temporally ordered speaker transcripts.Speaker-specific special tokens are inserted into the unified stereo transcript.
- 2.2 Architecture: DAC-VAE represents 48 kHz audio as compact 25 Hz, 128-dimensional latent features that support audio prompting and latent diffusion.The architecture uses partially masked noised audio features together with an audio context.
- 2.2 Architecture: Text-AB directly consumes raw text through mT5 embeddings and cross-attention, while frame-level language embeddings support multilingual generation.Unlike Audiobox, it does not rely on force-aligned text tokens.
- 2.2 Architecture: The model masks context regions during training and inference to handle mismatches in predicted audio context and language identities.This prevents differing context and target languages from creating an inference-time discrepancy in language-ID embeddings.
3 Training
Training progresses from multilingual monologue pretraining to specialized fine-tuning for dubbing, full-duplex dialogue, and explicit emotional control. The pipeline adapts the mono model for stereo dialogue and conditions emotion at the turn level.
- Pretraining: Pretraining uses 480k hours of multilingual speech and masked DAC-VAE feature infilling before downstream specialization.The data comprise 380k hours of English and 100k hours of Spanish.
- Training pipeline: The training branches from the pretrained mono model into dubbing, dialogue, and emotional-dialogue fine-tuning stages.Dialogue and emotional dialogue use separate downstream specializations after dubbing SFT.
- Dubbing SFT: Dubbing fine-tuning bridges inference conditions with sentence-level masking and 2.1k hours of mixed monolingual and synthetic cross-lingual data.The cross-lingual component provides 50 hours of voice- and style-aligned English–Spanish data.
- Dialogue SFT: Dialogue fine-tuning initializes Text-AB-Stereo from Text-AB-Mono and trains on 28k hours of English two-channel full-duplex conversations.Only the input and output projection layers are modified to accommodate two-channel speech.
- Emotional dialogue SFT: Emotional dialogue fine-tuning adds turn-level emotion style embeddings derived from valence, arousal, and dominance representations.The conditioning enables generated emotion to differ from the reference speaker’s tone while preserving voice identity.
4 Inference
Text-AB supports one-shot speech generation and arbitrarily long-form synthesis through multi-diffusion, with automated reranking to select higher-quality candidates.
- One-Shot Inference: Up to about one minute of speech can be generated in one shot, but quality degrades notably beyond that length.
- One-Shot Inference: Different audio-prompt configurations support empty prompts, mono prompts, and two-channel generation with prompted voice reuse.
- Long-Form Inference: Arbitrarily long speech is generated by independently processing overlapping chunks and merging their overlapping predicted flows with weighted averages.This multi-diffusion scheme addresses audible artifacts and discontinuities caused by concatenating separate one-minute dialogues.
- Multi-Stage Reranking: Multi-stage reranking selects among random-seed candidates using speaker similarity and word error rate, with additional turn-alignment and emotion metrics for emotional dialogue.
5 Experiment
Experiments evaluate Text-AB across dubbing, full-duplex dialogue, and emotional dialogue, showing gains from scaling, supervised fine-tuning, reranking, and emotion conditioning.
- Voice Dubbing Evaluation: SFT improves all dubbing objective metrics, while adding cross-lingual data further improves content accuracy and audio quality but reduces speaker similarity.
- Voice Dubbing Evaluation: 32 reranking candidates reduce dubbing WER from 4.05% to 2.20% and increase SpkSim from 0.66 to 0.74 versus no reranking.
- Full-Duplex Dialogue Synthesis Evaluation: Text-AB nearly matches real recordings on short-form dialogue, with only a −0.09 average human-likeness gap, and significantly outperforms the internal model on long-form human-likeness.Long-form gains extend to all fine-grained dimensions except NSVs and fillers.
- Full-Duplex Dialogue Synthesis Evaluation: A 30 s chunk size with 20 s overlap offers the best multi-diffusion trade-off across WER, Aes, speaker similarity, and real-time factor.More ODE steps and reranking candidates generally improve WER and Aes but increase real-time factor.
- Emotional Full-Duplex Dialogue Synthesis Evaluation: Emotional dialogue SFT consistently outperforms dialogue SFT in objective and subjective evaluations, improving alignment between intended and generated emotions.
- Emotional Full-Duplex Dialogue Synthesis Evaluation: Explicit emotion conditioning improves dialogue naturalness under emotional settings across LLM-based and human evaluations.With synthetic emotional text dialogues, the models achieve the top two scores under both evaluation protocols, while emotional dialogue SFT attains the highest human score.
6 Related Work
Related work spans zero-shot TTS, voice dubbing, full-duplex dialogue synthesis, and emotional speech synthesis, with prior methods differing in representations, interaction modeling, and emotion control.
- Zero-Shot TTS: Zero-shot TTS research includes codec-token language models and non-autoregressive discrete or continuous-latent generative approaches.
- Voice Dubbing: Voice dubbing typically uses cascaded ASR, machine translation, and TTS, while the TTS component remains a central bottleneck for preserving speaker attributes.
- Full-Duplex Dialogue Synthesis: Concatenating independently generated monologue turns often produces unnatural interactions, weak turn coordination, and limited conversational control.
- Full-Duplex Dialogue Synthesis: Most prior systems are single-channel, whereas stereo full-duplex dialogue is important for producing high-quality training data for full-duplex speech language models.
- Emotional Speech Synthesis: Emotional speech synthesis has used categorical labels, reference style embeddings, continuous intensity variables, and diffusion-based interpolation.
7 Conclusion
Text-AB unifies single- and two-channel speech generation in an alignment-free, flow-matching DiT framework, combining scalable training with flexible inference. It improves production voice dubbing and full-duplex dialogue synthesis while supporting arbitrarily long-form audio.
- The framework unifies single- and two-channel speech generation within one flow-matching DiT architecture while conditioning directly on raw text.
- A 3B-parameter model pretrained on multilingual monologues and fine-tuned for dubbing, dialogue, and emotional dialogue generalizes across monolingual, cross-lingual, and two-channel settings.
- Multi-stage reranking improves generation quality, while multi-diffusion extends one-shot synthesis to arbitrarily long-form audio with seamless transitions.
- Text-AB substantially outperformed the internal dubbing model and improved long-form dialogue human-likeness and expressivity, while approaching real recordings on short-form conversations.