Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis

Motivation

• The framework addresses cross-lingual dubbing and full-duplex dialogue, where preserving speaker characteristics across languages and controlling conversational content remain challenging.Existing dialogue pipelines also limit fluidity by stitching single-turn monologues algorithmically.Source: §1 · pp. 1–2

Scaling and training

• The system scales to a 3B-parameter model pretrained on 480k hours, followed by task-specific fine-tuning for dubbing and dialogue synthesis.The evaluated downstream tasks include voice dubbing, full-duplex dialogue synthesis, and emotional dialogue synthesis.Source: §1 · p. 2

Voice dubbing

• Text-AB substantially outperforms the latest internal dubbing model, gaining +0.39 shareability, +0.34 prosody similarity, +0.32 voice similarity, and +0.42 voice naturalness.These human-evaluation gains use a [−3, 3] MOS scale.Source: §1 · p. 2

Inference and dialogue modeling

• Text-AB supports one-shot generation up to about one minute and arbitrarily long-form generation through multi-diffusion.The dialogue model directly learns turn dynamics, back-channeling, and emotional evolution rather than relying on heuristic stitching or insertion.Source: §1 · pp. 2–3

Full-duplex dialogue synthesis

• Text-AB nearly matches real recordings on short-form dialogue and improves long-form human-likeness over the previous system by +0.86 MOS.The short-form gap is −0.09 overall human-likeness on a 5-point MOS scale.Source: §1 · pp. 2–3

2.1 Flow-Matching

• Flow matching trains the model to predict the velocity that moves a noised latent sample toward the target audio distribution.At inference, an ODE solver integrates the predicted velocity from Gaussian noise to generate the target latent.Source: §2.1 · p. 3
• Inference begins from Gaussian noise and solves an ODE with a fixed first-order Euler schedule to recover the generated latent.The number of time steps is tailored to the model.Source: §2.1 · p. 3

2.2 Architecture

• Text-AB-Mono and Text-AB-Stereo generate single- and two-channel speech, with stereo modeling jointly predicting both channels from temporally ordered speaker transcripts.Speaker-specific special tokens are inserted into the unified stereo transcript.Source: Figure 1 · p. 3, §2.2 · p. 4
• DAC-VAE represents 48 kHz audio as compact 25 Hz, 128-dimensional latent features that support audio prompting and latent diffusion.The architecture uses partially masked noised audio features together with an audio context.Source: §2.2 · p. 4
• Text-AB directly consumes raw text through mT5 embeddings and cross-attention, while frame-level language embeddings support multilingual generation.Unlike Audiobox, it does not rely on force-aligned text tokens.Source: §2.2 · p. 4
• The model masks context regions during training and inference to handle mismatches in predicted audio context and language identities.This prevents differing context and target languages from creating an inference-time discrepancy in language-ID embeddings.Source: §2.2 · p. 4

Pretraining

• Pretraining uses 480k hours of multilingual speech and masked DAC-VAE feature infilling before downstream specialization.The data comprise 380k hours of English and 100k hours of Spanish.Source: §3.1 · p. 5

Training pipeline

• The training branches from the pretrained mono model into dubbing, dialogue, and emotional-dialogue fine-tuning stages.Dialogue and emotional dialogue use separate downstream specializations after dubbing SFT.Source: §3.1 · p. 5

Dubbing SFT

• Dubbing fine-tuning bridges inference conditions with sentence-level masking and 2.1k hours of mixed monolingual and synthetic cross-lingual data.The cross-lingual component provides 50 hours of voice- and style-aligned English–Spanish data.Source: §3.2 · p. 5

Dialogue SFT

• Dialogue fine-tuning initializes Text-AB-Stereo from Text-AB-Mono and trains on 28k hours of English two-channel full-duplex conversations.Only the input and output projection layers are modified to accommodate two-channel speech.Source: §3.3 · p. 5

Emotional dialogue SFT

• Emotional dialogue fine-tuning adds turn-level emotion style embeddings derived from valence, arousal, and dominance representations.The conditioning enables generated emotion to differ from the reference speaker’s tone while preserving voice identity.Source: §3.4 · pp. 5–6

One-Shot Inference

• Up to about one minute of speech can be generated in one shot, but quality degrades notably beyond that length.Source: §4 · p. 6
• Different audio-prompt configurations support empty prompts, mono prompts, and two-channel generation with prompted voice reuse.Source: §4 · p. 6, Figure 2 · p. 6

Long-Form Inference

• Arbitrarily long speech is generated by independently processing overlapping chunks and merging their overlapping predicted flows with weighted averages.This multi-diffusion scheme addresses audible artifacts and discontinuities caused by concatenating separate one-minute dialogues.Source: §4 · p. 6, Figure 3 · p. 7

Multi-Stage Reranking

• Multi-stage reranking selects among random-seed candidates using speaker similarity and word error rate, with additional turn-alignment and emotion metrics for emotional dialogue.Source: §4 · pp. 6–7

Voice Dubbing Evaluation

• SFT improves all dubbing objective metrics, while adding cross-lingual data further improves content accuracy and audio quality but reduces speaker similarity.Source: §5.2 · p. 9, Table 3 · p. 9
• 32 reranking candidates reduce dubbing WER from 4.05% to 2.20% and increase SpkSim from 0.66 to 0.74 versus no reranking.Source: §5.2 · p. 9, Figure 4 · p. 10

Full-Duplex Dialogue Synthesis Evaluation

• Text-AB nearly matches real recordings on short-form dialogue, with only a −0.09 average human-likeness gap, and significantly outperforms the internal model on long-form human-likeness.Long-form gains extend to all fine-grained dimensions except NSVs and fillers.Source: §5.3 · pp. 9–10, Table 4 · p. 10
• A 30 s chunk size with 20 s overlap offers the best multi-diffusion trade-off across WER, Aes, speaker similarity, and real-time factor.More ODE steps and reranking candidates generally improve WER and Aes but increase real-time factor.Source: §5.3 · p. 10, Figure 6 · p. 11

Emotional Full-Duplex Dialogue Synthesis Evaluation

• Emotional dialogue SFT consistently outperforms dialogue SFT in objective and subjective evaluations, improving alignment between intended and generated emotions.Source: §5.4 · p. 11
• Explicit emotion conditioning improves dialogue naturalness under emotional settings across LLM-based and human evaluations.With synthetic emotional text dialogues, the models achieve the top two scores under both evaluation protocols, while emotional dialogue SFT attains the highest human score.Source: §5.4 · pp. 11–12

Zero-Shot TTS

• Zero-shot TTS research includes codec-token language models and non-autoregressive discrete or continuous-latent generative approaches.Source: §6 · p. 12

Voice Dubbing

• Voice dubbing typically uses cascaded ASR, machine translation, and TTS, while the TTS component remains a central bottleneck for preserving speaker attributes.Source: §6 · pp. 12–13

Full-Duplex Dialogue Synthesis

• Concatenating independently generated monologue turns often produces unnatural interactions, weak turn coordination, and limited conversational control.Source: §6 · p. 13
• Most prior systems are single-channel, whereas stereo full-duplex dialogue is important for producing high-quality training data for full-duplex speech language models.Source: §6 · p. 13

Emotional Speech Synthesis

• Emotional speech synthesis has used categorical labels, reference style embeddings, continuous intensity variables, and diffusion-based interpolation.Source: §6 · p. 13

• The framework unifies single- and two-channel speech generation within one flow-matching DiT architecture while conditioning directly on raw text.Source: §7 · p. 13
• A 3B-parameter model pretrained on multilingual monologues and fine-tuned for dubbing, dialogue, and emotional dialogue generalizes across monolingual, cross-lingual, and two-channel settings.Source: §7 · p. 13
• Multi-stage reranking improves generation quality, while multi-diffusion extends one-shot synthesis to arbitrarily long-form audio with seamless transitions.Source: §7 · p. 14
• Text-AB substantially outperformed the internal dubbing model and improved long-form dialogue human-likeness and expressivity, while approaching real recordings on short-form conversations.Source: §7 · p. 14