TBDub: Production-Oriented Visual Dubbing

• Visual dubbing must express replacement phonetics while preserving the source outside speech-related regions, where small identity or oral-detail changes can become conspicuous across video.Production inputs add challenges including occlusions, large poses, rapid motion, shot cuts, upstream generated content, and speech-segment junctions.Source: §1 · pp. 1–2
• Post-training adapts X-Dub to production data and failure modes with calibrated data curation, asymmetric lighting and oral degradation, an identity-matched clean oral prior, and multi-layer HuBERT conditioning.The Teacher also uses motion dropout, spatial face and mouth weighting, and later-frame and cold-start temporal weighting.Source: §1 · p. 2
• Few-step distillation extends DMD/DMD2 to visual dubbing using X-Dub’s three-route guidance, a differentiable two-step video trajectory, staged ground-truth warmup, and region-selective supervision.These adaptations target stable identity and oral detail during compression.Source: §1 · p. 2
• The two-step Student reaches 7.13 FPS, with 13.93× end-to-end and 42.49× DiT speedups over the 30-step Teacher while largely retaining its quality.The reported production-related improvements also include stronger qualitative robustness, visual quality, identity preservation, and audiovisual synchronization.Source: §1 · p. 2

2.1 Task Definition

• Mouth motion should follow replacement speech while non-speech content remains consistent with the source video.The task balances audiovisual response, visual preservation, and temporal naturalness, and has no unique frame-aligned ground truth for replacement audio.Source: §2.1 · p. 2

2.2 Inherited X-Dub Structure

• X-Dub uses a video diffusion model with full reference-video context and no inference-time mouth mask, but its editable face and scene context can permit unwanted changes.Its inherited stack jointly models source-reference and target latents and uses conditional speech features and classifier-free guidance.Source: §2.1 · pp. 2–3, §2.2 · p. 3, Table 1 · p. 3

2.3 Production Failure Modes

• Production failures include identity-sensitive texture drift, loss or flicker of small-face oral detail, and copying oral artifacts from generated references.Examples include blurred or fused teeth, green tooth color, smeared texture, and lip–tooth adhesion.Source: §2.3 · p. 3
• Production inputs also expose illumination and motion discontinuities, weak robustness to difficult scenes, speech-segment junction failures, and the throughput cost of 30 sampling steps.The reported symptoms include flicker, trajectory drift, boundary jitter, delayed or discontinuous mouth motion, and, in severe cases, failed segments.Source: §2.3 · p. 4

3.1 Production-Domain Extension of the X-Dub Pseudo-Pair Pipeline

• Pseudo-pairs use a redubbed reference and the clean source as target, preserving subject and scene context while changing speech-related mouth motion.Identity, articulation, and visual-quality filters remove unsuitable pairs; production-specific checks add crop, blur, flicker, mouth-jitter, and decoding-failure screening.Source: §3.1 · p. 4

3.2 Controlled Production-Domain Data Mixture

• Livestream and green-screen data are sampled at a 3:1 ratio to combine difficult production conditions with clean oral-detail supervision.The two sources complement one another: livestreams cover practical failures, while green-screen recordings provide stable illumination and clean lip and tooth texture.Source: §3.2 · pp. 4–5

3.3 Conditioning-Only Asymmetric Relighting

• Conditioning-only relighting, color-temperature changes, and background replacement train recovery of the unchanged clean target from varied reference appearance.Augmentation is sampled consistently within a clip to avoid artificial frame-wise flicker as a training cue.Source: §3.3 · p. 5

3.4 Oral Degradation Modeling

• Oral degradation corrupts only the conditioning reference’s mouth region while supervision remains the corresponding clean frames.Degradation strength and noise are shared across a clip, avoiding unrealistic frame-level corruption patterns.Source: §3.4 · p. 5

3.5 Identity-Matched Clean Oral Prior

• An identity-matched clean oral prior supplies stable lip, tooth, and oral texture, while the dynamic reference provides pose and scene context and audio determines articulation.Candidate frames are selected using mouth geometry and local visual quality.Source: §3.5 · p. 5

3.6 Segment-Aligned Multi-Layer HuBERT Conditioning

• The audio frontend concatenates HuBERT-large layers 9–12 and aligns local feature windows to video latent time, retaining nearby phonetic context across pauses and chunk transitions.Teacher, Student, and FakeScore use the same aligned feature representation during distillation.Source: §3.6 · pp. 5–6

3.7 Post-Trained Teacher

• The 30-step Teacher uses spatial and temporal flow-matching weights that emphasize face and mouth tokens, later frames, and cold-start first frames without motion context.The post-trained model combines these weights with production-domain data, conditioning changes, a clean oral prior, and multi-layer HuBERT features.Source: §3.7 · p. 6

4 Task-Aware Few-Step Distillation

• The 30-step Teacher’s cost exceeds the target deployment budget, motivating compression to a Student that must still preserve identity, scene, audio response, motion context, and facial detail.The challenge is to retain these editing constraints after a drastic reduction in numerical integration steps.Source: §4 · p. 6

4.1 From DMD and DMD2 to Conditional Video Editing

• DMD2 distillation for visual dubbing requires a guided real score, a differentiable two-step conditional trajectory, and region-selective reconstruction constraints.DMD uses a Teacher–FakeScore gradient; DMD2 adds alternating optimization and multi-step generation, while its optional adversarial branch is disabled in the final configuration.Source: §4.1 · pp. 6–7

4.2 Reusing X-Dub’s Native Guidance for the DMD Real Score

• The DMD real score combines X-Dub’s nested reference and audio routes, separating reference preservation from audio’s incremental effect given the reference.Guidance scales vary with noise level; the first temporal latent instead uses a fixed reference scale of one to avoid over-guidance in the causal VAE.Source: §4.2 · p. 7

4.3 Differentiable Two-Step Video Generation within DMD2

• The Student’s two-step conditional video trajectory retains its complete computation graph, so final-latent distribution matching trains both steps together.A single fully conditioned route replaces the Teacher’s three branches at deployment, with consistent conditions supplied to the Student, Teacher, and FakeScore.Source: §4.3 · pp. 7–8

4.4 Conditional Distribution-Matching Objective

• Conditional distribution matching perturbs the Student’s final latent and compares guided Teacher and online FakeScore predictions under shared conditions.Known motion-context positions are excluded from the generation loss, and sampling spans the full noise range, including low noise.Source: §4.4 · p. 8

4.5 Ground-Truth Warmup and Region-Selective Supervision

• A finite real-data warmup stabilizes early distillation, while selective weighting excludes the eyes and most of the lower face and retains light lip supervision.The warmup uses ground-truth latent MSE for the first 2,000 Student updates with lip weight 0.1, then is removed.Source: §4.5 · pp. 8–9

4.6 Optimization Configuration

• Training alternates five FakeScore updates per Student update and reports the two-step Student using FP32 EMA-smoothed weights.Optional adversarial, perceptual, identity, and temporal objectives are disabled in the final configuration.Source: §4.6 · p. 9

4.7 Task Adaptations beyond DMD/DMD2

• TBDub connects X-Dub’s guided score to DMD, keeps conditions consistent across the Student, Teacher, and FakeScore, and adds full-trajectory gradients, motion-context exclusion, and region-selective supervision.Causal first-latent handling, full-timestep sampling, FP32 guidance arithmetic, and FP32 EMA are additional implementation adaptations.Source: §4.7 · p. 9

5.1 Evaluation Protocol

• The evaluation uses 38 TalkVid background clips, with matching-audio self-driven tests and replacement-audio cross-driven tests; six qualitative categories and randomized, anonymized MOS ratings broaden the comparison.Objective metrics use a common 512 × 512 crop protocol, method-specific frame maps, and per-video macro-averaging for most metrics.Source: §5.1 · p. 10

5.2 Metrics

• Evaluation combines frame-aligned reconstruction and perceptual measures, identity similarity, SyncNet alignment, response magnitude, and MOS dimensions for synchronization, identity, and visual quality.Response Mouth PSNR is a response-magnitude diagnostic, not a quality score; SyncNet measures are treated as diagnostic rather than absolute human-perception measures.Source: §5.2 · pp. 10–11

5.3 Objective Comparison

• Across 38 TalkVid clips, the TBDub Teacher moves all eight reconstruction, perceptual, identity, and SyncNet metrics in the preferred direction relative to X-Dub.Response Mouth PSNR rises from 23.05 to 25.67, indicating stronger source-appearance preservation but a more conservative response to replacement audio.Source: Table 2 · p. 11, §5.3 · p. 11
• The two-step Student largely retains the Teacher’s visual quality, identity consistency, and synchronization; CSIM remains 0.91 while LSE-C rises from 6.80 to 7.16.Mouth PSNR decreases from 27.13 to 26.34, while FID decreases from 9.40 to 9.10.Source: §5.3 · p. 11, Table 2 · p. 11

5.4 Subjective MOS Evaluation

• The Student has the highest lip-sync and visual-quality MOS means, 3.85 and 3.80, while the Teacher has the highest identity-consistency mean, 3.78.Compared with X-Dub, the Teacher gains 0.14 in lip-sync consistency, 0.95 in identity consistency, and 0.90 in visual quality.Source: Table 3 · p. 12, §5.4 · p. 12

5.5 Qualitative Comparison

• Across language variation, generated-video domain shift, pose, occlusion, motion, and uncontrolled capture, video comparisons show stable oral structure and appearance with reduced lip flicker, fused teeth, and local identity drift.Source: §5.5 · p. 13

6.1 End-to-End Generation Throughput

• 13.93×: the two-step Student reaches 7.13 effective FPS, approximately 13.93× the TBDub Teacher’s throughput under the stated benchmark.The benchmark measures generation from the first VAE encode through the final VAE decode on one NVIDIA H20 at 512 × 512.Source: Table 4 · p. 13, §6.1 · p. 13

6.2 Paired Stage-Wise Profile

• 13.93×: Student latency falls from 734.75 to 52.76 seconds, while DiT execution falls from 696.77 to 16.40 seconds, a 42.49× speedup.The DiT gain reflects both fewer sampling steps and replacing the Teacher’s three-route guidance evaluations with one fully conditioned Student route per step.Source: Table 5 · p. 14, §6.2 · p. 14
• 44.61%: final VAE decoding is the largest Student latency component, compared with DiT execution at 31.09%.DiT accounts for 94.83% of Teacher latency, so fixed codec and pipeline costs limit the end-to-end speedup.Source: §6.2 · p. 14
• The Student’s throughput gain comes with only small decreases in Mouth PSNR and CSIM, while remaining perceptual, distribution, and synchronization metrics remain comparable or improve slightly.The paired profile and quality results support a more favorable quality–efficiency operating point for the Student.Source: §6.2 · p. 14

7 Limitations

• Severe compression, blur, or very few facial pixels can limit recovery of identity-sensitive lip and tooth details; a separate face super-resolution stage is outside the current pipeline.The system remains reference-conditioned, so output quality is bounded by input quality and spatial resolution.Source: §7 · p. 14
• Full-reference reconstruction metrics apply only to self-driven evaluation, while SyncNet metrics have limited, domain-dependent correlation with human perception.The paper recommends interpreting cross-driven quality jointly through objective diagnostics, MOS, and qualitative video comparisons.Source: §7 · p. 14

8 Conclusion

• TBDub develops task-adaptive post-training and two-step distillation on the X-Dub backbone, using production-domain adaptation and task-specific methods for conditional video editing.The Student incorporates a differentiable two-step trajectory, motion-state exclusion, staged region-selective supervision, causal first-latent handling, and FP32 EMA.Source: §8 · pp. 14–15
• On 38 TalkVid clips, the TBDub Teacher improves all eight reported reconstruction, perceptual, identity, and SyncNet metrics over X-Dub.In the MOS study, the Teacher improves lip-sync consistency, identity consistency, and visual quality over X-Dub by 0.14, 0.95, and 0.90 points, respectively.Source: §8 · p. 15
• The two-step Student largely preserves the Teacher’s generation quality and audiovisual synchronization while reducing measured end-to-end latency by 13.93× and DiT latency by 42.49×.The Student reaches 7.13 effective FPS; its MOS lip-sync and visual-quality means are highest, and its identity-consistency score trails the Teacher by 0.06.Source: §8 · p. 15