PaperlayerView exact revision on arXiv

Source-linked AI summary

Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing

Rui Liu, Bhavin Jawade, Haoqi Li, Shivam Mehta, Karan Saxena, Yinghong Lan, Cameron R. Wolfe

arXiv:2610.00825v1cs.CV

TL;DR

A reference-free dubbing judge must assess whether candidate text matches visible articulation in both content and timing, even before dubbed audio exists. ATR aligns mouth motion with candidate phonetic units, then reasons over local and global alignment evidence; it improves benchmark performance and transfers across models, languages, and dubbing tasks.

  • Problem

    Dubbing review needs a judge that checks candidate text against silent video for both phonetic compatibility and temporal order, while existing visual-speech and video-language models are poorly suited to that task.

    Source: §1 · pp. 1, 3

  • Method

    ATR uses candidate-conditioned monotonic alignment between frame-level lip representations and phonetic units, then gives an LLM per-unit soft tokens and a calibrated global score for its judgment.

    Source: §1 · p. 2, §2.3 · p. 4

  • Results

    Across seven languages, ATR improves mean AUC over corresponding Qwen3.5 SFT baselines by 59.4%, 50.2%, and 50.8% with 2B, 4B, and 9B reasoners, respectively, with gains also reported across LLM families, unseen languages, and downstream dubbing tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results support explicit monotonic alignment as a source of temporal grounding alongside LLM content discrimination and grounded explanations.

    Source: §1 · p. 3

Abstract

from arXiv · show

Dubbing quality control requires a reference-free judge that can determine whether a candidate text line matches a speaker's visible articulation in both content and timing, using only silent video and text because dubbed audio may not yet exist. Existing visual speech recognizers and video-language models are poorly suited to this setting: even when fine-tuned to recover spoken content from lip motion, they remain largely insensitive to temporal errors. We introduce $\textit{Align Then Reason}$ (ATR), a multilingual lip-sync judge that first establishes a monotonic alignment between frame-level lip representations and the phonetic units of the candidate line, then reasons over this alignment to make the final judgment. An alignment scorer provides the LLM with both local evidence for each phonetic unit and a calibrated global alignment score, enabling it to reason jointly about content and timing. On a seven-language benchmark, our method improves mean AUC over the corresponding Qwen3.5 SFT baselines by 59.4%, 50.2%, and 50.8% with 2B, 4B, and 9B reasoners, respectively. The gains generalize across LLM families, reaching mean AUC improvements of 45.9% and 46.6% over the best baseline for LLaMA-3.1-8B and Mistral-7B, respectively. They also transfer across datasets to three unseen MuAViC languages. Furthermore, we evaluate on two downstream tasks built from real dubbing lines. On dub-line reranking, ATR-9B outperforms the best lip-reading baseline by 52.0%, while on script-to-clip assignment, ATR-9B improves over the best lip-reading baseline by 17.7%.

1 INTRODUCTION

Dubbing review needs a reference-free judge that checks whether candidate text matches visible articulation in both content and timing. ATR addresses temporal weaknesses in existing models by aligning phonetic units with mouth motion before LLM reasoning.

1 INTRODUCTION

  • A useful judge must work from silent video and candidate text, detecting both incompatible phonetic content and incorrect timing or temporal order across languages.A plausible translated line can preserve meaning yet start too early, lag behind the mouth, or conflict with visible articulation.Source: §1 · p. 1
  • Existing visual speech recognizers target transcription, audiovisual synchronization models require audio, and video-language models do not guarantee temporal reasoning.With candidate text fixed, Qwen3.5 models from 2B to 9B remain near chance when video frames are reversed, shifted, or frozen.Source: §1 · pp. 1–2
  • ATR aligns frame-level lip motion monotonically with candidate phonetic units, then gives an LLM per-unit soft tokens and a calibrated global score for its judgment.The candidate-conditioned CTC scorer is trained with content and temporal negatives, and the reasoner returns a binary judgment and grounded explanation.Source: §1 · pp. 2–3
  • Across seven languages, ATR improves mean AUC over corresponding Qwen3.5 SFT baselines by 59.4%, 50.2%, and 50.8% with 2B, 4B, and 9B reasoners, respectively.It also transfers across LLM families and to three unseen MuAViC languages; on real dubbing lines, it beats the strongest lip-reading baseline by 52.0% on dub-line reranking and 17.7% on script-to-clip assignment.Source: §1 · p. 3
  • ATR couples temporal tracking to semantic evaluation, anchoring LLM reasoning in fine-grained alignment dynamics.The paper identifies temporal insensitivity in autoregressive visual-speech and video-language models as a structural gap for text-video lip-sync judgment.Source: §1 · p. 3

2 APPROACH

ATR judges whether a candidate line matches visible mouth movements in content and timing, without dubbed audio or a reference transcript. It aligns frame-level lip representations with candidate phonetic units, then gives local and global alignment evidence to an LLM reasoner.

2.1 PROBLEM FORMULATION

  • The judge takes silent video and a candidate line, then predicts whether their content and timing agree without dubbed audio or a reference transcript.Source: §2.1 · p. 3

2.2 CROSS-MODAL MONOTONIC ALIGNMENT

  • Frame-level lip and phonetic-unit embeddings yield compatibility scores, and candidate-position CTC enforces a monotonic alignment through the units.Source: §2.2 · p. 3
  • A length-normalized CTC score allows phonetic units to span consecutive frames and blank frames between units while preserving their original order.Source: §2.2 · p. 4
  • Contrastive training compares genuine pairs with content- and timing-corrupted alternatives, while auxiliary CTC supervision trains visual embeddings to preserve phonetic information.Source: §2.2 · p. 4

2.3 LLM REASONER

  • The LLM receives candidate text, per-unit representations that retain local alignment evidence, and a calibrated global score for the whole line.Source: §2.3 · p. 4
  • At inference, the frozen alignment scorer supplies soft tokens and a calibrated score to the reasoner, whose final score is the Yes-versus-No logit difference.Source: §2.3 · pp. 4–5

3 EXPERIMENTS

The experiments test ATR across languages, model backbones, corruption types, and dubbing tasks. Results show strong benchmark performance and transfer, with ablations linking its components to content or timing evidence.

3.1 EXPERIMENTAL SETUP

  • The benchmark has approximately 50K multilingual training samples and 2,100 test samples across seven languages, with content and temporal corruptions evaluated against broad recognition and vision-language baselines.Content negatives include mismatches, word-order shuffles, and dub lines; temporal negatives reverse, shift, freeze, or swap frames. The primary metric is pooled AUC, with paired accuracy also reported.Source: §3.1 · p. 5, §3.2 · p. 5

3.2 MAIN RESULTS

  • 0.920 mean AUC with Qwen3.5-9B is a 50.8% relative improvement over the strongest baseline mean of 0.610 on the seven-language benchmark.+0.310 absolute AUC.Source: §3.2 · p. 6, Table 1 · p. 6
  • ATR paired accuracy exceeds 0.916 across model variants, while frontier models average 0.656–0.737 on temporal corruptions.Source: §3.2 · p. 6, Table 2 · p. 7

3.3 GENERALIZATION TESTING

  • On three unseen MuAViC languages, recalibrated ATR-9B reaches 0.875 mean AUC, improving 70.2% over Qwen3.5-9B Base and 51.4% over Auto-AVSR.Recalibration fits two scalar normalization parameters using 300 held-out genuine samples without model-weight updates; freeze AUC rises from 0.572 to 0.937.Source: §3.3 · pp. 6–7, Table 3 · p. 7
  • ATR also transfers to LLaMA-3.1-8B and Mistral-7B, reaching mean AUCs of 0.890 and 0.894, respectively.Source: §3.3 · p. 7

3.4 ABLATION STUDIES

  • Ablations associate phonetic supervision and the reasoner with content discrimination, soft tokens with local evidence, and the calibrated scalar with temporal sensitivity.Removing phonetic supervision, soft tokens, or the calibrated scalar reduces mean AUC from 0.920 to 0.898, 0.872, or 0.672, respectively; removing the scalar most weakens temporal discrimination.Source: §3.4 · pp. 7–8, Table 4 · p. 8

3.5 DOWNSTREAM EVALUATION

  • On dub-line reranking, ATR-9B achieves 0.479 Top-1 accuracy versus 0.316 for the strongest lip-reading baseline, a 52% relative improvement.Each test sample pairs a professional dub with three meaning-preserving, length-matched paraphrases.Source: §3.5 · p. 8, Table 5 · p. 8
  • For in-domain script-to-clip assignment, ATR-9B achieves 0.746 per-clip accuracy, 17.7% higher relative to Auto-AVSR, and 0.522 Exact Block accuracy.For MuAViC two-way assignment, ATR reaches 0.704 per-clip accuracy versus 0.598 for Auto-AVSR.Source: §3.5 · p. 9

4 RELATED WORK

Prior approaches address pieces of dubbing alignment but do not directly score arbitrary candidate text against silent lip motion. ATR combines candidate-conditioned visual-phonetic alignment with language-model reasoning over alignment evidence.

  • Prior dubbing systems use duration, prosody, and isochrony constraints, but these do not assess phonetic compatibility with visible articulation [Federico et al., 2020; Tam et al., 2022].Source: §4 · p. 9
  • In script-to-clip assignment, ATR-9B reaches 0.746 per-clip accuracy in-domain, 17.7% relatively above Auto-AVSR.On MuAViC two-way pairs, ATR scores 0.704 versus 0.598 for Auto-AVSR.Source: Table 6 · p. 9
  • Visual speech recognition, audio-based synchronization, and visual forced alignment each miss reference-free candidate scoring from silent video [Shi et al., 2022; Ma et al., 2023; Yeo et al., 2024; Cappellazzo et al., 2025; Prajwal et al., 2020; Kim et al., 2023].ATR instead learns candidate-conditioned visual–phonetic alignment without alignment labels, using monotonic CTC to score content compatibility and temporal order.Source: §4 · p. 9
  • Rather than asking an LLM to infer timing from video, ATR supplies local alignment evidence and a calibrated global score for higher-level content reasoning [Buch et al., 2022; Lei et al., 2023; Liu et al., 2024; Li et al., 2025a].The calibrated scorer output forms the interface between temporal grounding and language reasoning.Source: §4 · p. 9

5 CONCLUSIONS

ATR is a reference-free multilingual lip-sync judge that combines visual phonetic alignment with LLM reasoning. It improves benchmark and downstream-task performance, with generalization to unseen languages.

5 CONCLUSIONS

  • ATR achieves up to a 50.8% mean AUC improvement with a 9B reasoner and generalizes to unseen languages.The reasoner is trained with supervised fine-tuning; reinforcement learning is identified as a natural next step.Source: §5 · p. 10
  • ATR-9B outperforms top lip-reading baselines by 52.0% on dub-line reranking and 17.7% on script-to-clip assignment.Source: §5 · p. 10
  • ATR is positioned as an automatic evaluation signal for systems that rewrite dub lines without human review.Source: §5 · p. 10

A EXPERIMENTAL SETUP

The visual encoder produces frame-level features that a temporal lip adapter maps into a 256-dimensional alignment space. Candidate phonetic units are separately encoded and projected into that space.

A EXPERIMENTAL SETUP

  • Auto-AVSR produces 768-dimensional frame features, which the temporal lip adapter maps to 256-dimensional lip embeddings.The adapter uses layer normalization, a linear projection, a two-layer temporal Transformer with eight attention heads, and a final projection.Source: A EXPERIMENTAL SETUP · p. 14
  • Candidate text is converted to phonetic units, encoded with XPhoneBERT, and projected from 768 to 256 dimensions.Source: A EXPERIMENTAL SETUP · p. 14
  • The LLM reasoner is adapted with LoRA at rank 16 and α = 32, and models are trained with AdamW at learning rate 3 × 10−4.The reported setup uses weight decay 0.01 and batch size 32.Source: A EXPERIMENTAL SETUP · p. 14

B REASONER PROMPT

The reasoner prompt includes the alignment confidence and candidate line, then asks whether the mouth movements match. Training uses a Yes/No answer with a grounded explanation; inference reads the answer-token logits.

B REASONER PROMPT

C ENGLISH-ONLY AND PER-LANGUAGE RESULTS

Across seven languages, the alignment scorer maintains strong content discrimination and near-ceiling temporal sensitivity. English-only results also show strong ATR-9B performance across most evaluation axes, while Auto-AVSR leads on mismatch.

C ENGLISH-ONLY AND PER-LANGUAGE RESULTS

D STATISTICAL SIGNIFICANCE

The significance evaluation uses three random seeds and reports mean AUC and standard deviation across the seven-language benchmark and two MuAViC evaluations. ATR shows low variance on the seven-language benchmark, while recalibration reduces out-of-domain variance and boosts mean accuracy.

D STATISTICAL SIGNIFICANCE

  • 0.910 ± 0.015 for Qwen3.5-9B and 0.895 ± 0.012 for LLaMA-3.1-8B show low variance on the seven-language benchmark across three seeds.Table 9 reports mean AUC and standard deviation for the seven-language benchmark, zero-shot MuAViC, and recalibrated MuAViC.Source: D STATISTICAL SIGNIFICANCE · p. 14, Table 9 · p. 15
  • 0.778 ± 0.060 for Qwen3.5-9B describes higher variance in uncalibrated MuAViC scores; two-parameter recalibration reduces variance and yields 0.852 ± 0.033 mean accuracy.The passage attributes the initial variance to domain shifts in scalar distributions.Source: D STATISTICAL SIGNIFICANCE · p. 14
  • On English-only pooled-AUC evaluation, ATR-9B outperforms all baselines across six content and temporal axes, while Auto-AVSR wins mismatch.The recognizer baselines were trained primarily on English data.Source: Table 7 · p. 15
  • Across seven languages, the alignment scorer remains at ≥0.96 AUC on temporal corruptions, while Auto-AVSR degrades to chance on Japanese and Korean.Auto-AVSR also shows weak temporal sensitivity on non-English languages.Source: Table 8 · p. 15
  • Across three seeds, Table 9 covers mean AUC and standard deviation for the seven-language benchmark, zero-shot MuAViC, and recalibrated MuAViC.The caption reports stable performance gains across diverse LLM reasoner backbones on these evaluations.Source: D STATISTICAL SIGNIFICANCE · p. 14, Table 9 · p. 15

Related papers

Loading 2610.00825v1…