PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Lip Sync: Reference-Grounded Oral Refinement for Audio-Driven Portrait Animation

Bangxun Tang

arXiv:2609.38019v1cs.CV

TL;DR

Lip-sync systems can follow audio and preserve recognizable faces while rendering a generic mouth rather than the person’s own lips and teeth. RGOR combines separate-recording references, high-definition mouth patches, and a paired judge trained to compare rendered mouths with the person’s reference. On held-out identities, it ranks first or second on every fidelity metric and preserves oral detail while leaving the rest of the face untouched.

  • Problem

    Current systems can synchronize speech and preserve face recognition, but their training and evaluation do not ensure that the rendered lips and teeth match the specific person.

    Source: §1 · p. 2

  • Method

    RGOR conditions frames on scattered enrollment references and high-definition mouth patches, and trains a paired judge to compare rendered mouths with the person’s reference.

    Source: §1 · p. 2

  • Results

    RGOR ranks first or second on every fidelity metric against open-source and commercial systems on the held-out protocol.

    Source: §6 · p. 10

  • Takeaways & Limitations

    RGOR preserves the person’s lip and dental detail while leaving the rest of the face untouched.

    Source: §6 · p. 10

Abstract

from arXiv · show

We present RGOR (Reference-Grounded Oral Refinement), an audio-driven lip-sync framework that renders the mouth of the specific person being dubbed rather than a generic one. Existing lip-sync systems follow the audio closely and keep the face recognizable, yet the mouth they render is an average mouth: the shape and texture of the lips, the arrangement of the teeth, and how much of them shows as the mouth opens are not that person's. The problem persists because nothing in current training or evaluation asks for the person's own mouth: perceptual losses accept any plausible mouth, face identity is carried mostly by the skin around it, and the released inference code of inpainting systems uses the unmasked target frame as the reference, which hides the gap. To address this, RGOR conditions every generated frame on frames from separate enrollment recordings of the same person and on HD patches of the mouth that bypass the VAE, and trains the generator against a paired judge that compares each rendered mouth with the person's reference and learns to reject a realistic mouth of someone else. We further build an evaluation protocol and use it to compare open-source and commercial lip-sync systems on held-out identities. Experiments show that RGOR achieves the best or second-best result on most metrics, and preserves the person's own lip and dental detail while keeping synchronization and the rest of the face intact.

1 INTRODUCTION

Current lip-sync systems can synchronize speech and preserve recognizable faces while rendering generic rather than person-specific lips and teeth. RGOR addresses this gap with same-person enrollment references, high-definition mouth patches, and training objectives that compare the rendered mouth with the reference.

1 INTRODUCTION

  • Existing systems can follow audio and keep faces recognizable, yet render generic lip texture, teeth, and mouth-opening patterns rather than the dubbed person’s own mouth.The introduction frames this mismatch as especially consequential because the mouth is where speech is seen.Source: §1 · p. 2
  • Training and evaluation often do not require the rendered mouth to match the person, and same-frame references in released inpainting pipelines can leak the target mouth into paired tests.With a reference from another recording of the same person, LatentSync and Wav2Lip mouth fidelity drops markedly (Appendix C).Source: §1 · p. 2
  • RGOR conditions each frame on scattered references from separate recordings and high-definition mouth patches, then uses a reference-contrastive paired judge and a shape term to target the person’s oral appearance.The references show varied mouth shapes without a motion to copy; the shape term supervises the opening and lip regions.Source: §1 · p. 2
  • The evaluation holds out identities and separates enrollment clips from query clips, with systems receiving a masked query or a different recording of the person.The protocol is designed to evaluate reproduction without exposing the target mouth as a reference.Source: §1 · p. 2
  • The paper’s contributions are RGOR, a paired oral-identity comparison with a shape term, and a protocol for testing reference leakage and comparing systems on unseen identities.The comparison includes open-source and commercial lip-sync systems.Source: §1 · p. 2

2 RELATED WORK

Prior lip-sync systems improve synchronization and image quality, but their objectives and evaluation do not ensure that the rendered mouth belongs to the specific speaker. RGOR adds separate-recording references and an objective that compares the generated mouth with those references.

2 RELATED WORK

  • Lip-sync systems progressed from inpainting with synchronization supervision to GAN and diffusion methods, while other approaches animate portraits or use mask-free editing and commercial services.Despite these advances, inpainting methods leave whose mouth is rendered unsupervised, and RGOR adds supervision using references from separate recordings.Source: §2 · pp. 2–3
  • Reference-based identity methods inject appearance or identity information into generators, but their objectives do not necessarily require outputs to use the specific person’s mouth.RGOR pairs references with an objective that compares the rendered mouth against them.Source: §2 · p. 3
  • Existing judges assess realism, synchronization, or target similarity, while RGOR’s judge scores whether two mouths belong to the same person.Its negative examples include real mouths of other people, so realism alone cannot satisfy the identity comparison.Source: §2 · p. 3
  • Common lip-sync metrics have mouth-specific blind spots, and same-frame references in released inpainting code can expose the reconstruction target during paired evaluation.NeRSemble’s separate recordings support an enrollment/query split for a cross-clip evaluation protocol.Source: §2 · p. 3
  • RGOR’s overview combines scattered enrollment references and high-definition mouth patches with image, shape, and paired-judge training signals.The patches bypass the VAE through gated attention, while masked source frames, references, and audio drive the inpainting UNet.Source: Figure 2 · p. 4

3 METHOD

RGOR builds on a latent inpainting generator and changes its references, adds high-resolution mouth inputs, and trains a person-specific paired judge alongside a geometric shape constraint. Its data and training setup use separate enrollment clips and composite-frame objectives.

3.1 OVERVIEW

  • The parent generator denoises aligned face-frame windows in a frozen VAE’s latent space, conditioned on audio, a lower-face mask, the masked source, and a reference frame.The decoded prediction is composited into the source through a feathered editable-region mask.Source: §3.1 · p. 4

3.2 SCATTERED ENROLLMENT REFERENCES

  • Each output frame receives a separate enrollment reference, selected with preference for visible teeth, so the references show varied mouth shapes without a motion sequence to copy.The training bank excludes query clips; inference selects sixteen bank frames in a fixed order.Source: §3.2 · p. 4

3.3 HD PATCHES

  • Four full-resolution mouth crops spanning closed to widest-open states bypass the VAE and reach the UNet through a reference encoder and gated attention adapters.This preserves fine lip and tooth structure that the latent-scale references would lose.Source: §3.3 · p. 5

3.4 REFERENCE-CONTRASTIVE PAIRED JUDGE

  • The paired judge compares rendered and reference mouth crops to decide whether they belong to the same person, with real mouths of other people serving as negatives.The generator is trained against the judge, so a realistic generic mouth does not satisfy the person-specific comparison.Source: §3.4 · p. 5

3.5 SHAPE TERM

  • A frozen face parser supplies mouth-interior and lip-region targets for a shape term that supervises opening and visible lip geometry.The Dice-based term constrains geometry without pulling texture toward an average.Source: §3.5 · pp. 5–6

3.6 DATA

  • Training data use 290 identities, with a disjoint 50-identity held-out benchmark, and build each sample from a masked window, its audio, enrollment references, mouth patches, and the unmasked target.The corpus is drawn from frontal-camera NeRSemble recordings.Source: §3.6 · p. 6

3.7 TRAINING AND INFERENCE

  • Training adds paired and shape losses to LPIPS, boundary, and DISTS image terms; inference denoises clips in 20 steps using fused overlapping 16-frame windows.The objectives act on the composite, with gradients through the frozen decoder.Source: §3.1 · p. 4, §3.7 · p. 6

4 EVALUATION PROTOCOL

The protocol evaluates held-out identities using separate enrollment and query clips, withholding the target mouth from systems that accept a separate reference. It distinguishes masked-input inpainting systems from cross-video systems that must re-render another clip.

4 EVALUATION PROTOCOL

  • The benchmark uses 50 unseen identities, each with two query sentences and two separate enrollment sentences, for 100 query clips.Inpainting systems receive masked target frames and same-person references from other clips; cross-video systems re-render another clip with the target audio.Source: §4 · p. 6
  • Except for MuseTalk-1.5 run as released, systems are not shown the target clip’s mouth; the released inpainting code’s same-frame reference is replaced for cross-reference evaluation.The alternate clip is recorded in the same session with the same frontal camera, making it close in head position and expression.Source: §4 · p. 6
  • Fidelity is measured within a fixed oral box, synchronization with LSE-C and LSE-D, and changes outside the mouth with MAE-out.Oral measures include pixel, perceptual, texture-ratio, and open-area-ratio metrics.Source: §4 · pp. 6–7

5 EXPERIMENTS

On the held-out benchmark, RGOR preserves person-specific lip and dental detail while maintaining synchronization and leaving the rest of the face intact. It performs strongly across automatic metrics and human judgments, and its ablations show that its components address distinct aspects of mouth fidelity.

5.1 EXPERIMENTAL SETUP

  • The comparison evaluates open-source, commercial, and mask-free systems using released weights or public APIs with recommended settings, changing only their inputs.The evaluation uses the held-out benchmark and protocol described in Section 4.Source: §5.1 · p. 7

5.2 RESULTS

  • RGOR reconstructs person-specific lip contour, texture, and teeth from enrollment references while keeping synchronization and the untouched face intact.Only the masked lower face is regenerated and composited back into the source video.Source: §5.2 · p. 7
  • RGOR achieves the best oral LPIPS and DISTS of all systems, and its oral DISTS is lower than every baseline on 95 of 100 query clips.It is a close second to LatentSync-1.6 on pixel-averaging metrics and scores better than ground truth on both LSE-C and LSE-D.Source: §5.3 · p. 8, §5.2 · p. 8

5.3 COMPARISON AND EVALUATION

  • On the same-frame comparisons, RGOR renders sharper, more person-specific teeth and lips than the baselines, with an opening that matches the ground truth.The inpainting baselines blur or genericize dentition, while X-Dub tends to open the mouth too wide.Source: §5.3 · p. 8

5.4 ABLATION STUDY

  • The full model is best or close to best on every ablation metric, and removing each component damages a distinct aspect of mouth fidelity.Contiguous references copy a short stretch; removing HD patches softens fine detail; removing the paired judge yields generic teeth; removing the shape term under-articulates the opening.Source: §5.4 · p. 9

5.5 HUMAN STUDY

  • Human raters prefer RGOR on all four criteria, with the strongest preference for mouth identity.The four criteria are mouth identity, the face outside the mouth, realism, and lip sync; Figure 6 summarizes vote shares.Source: §5.5 · p. 9, Figure 6 · p. 9

6 CONCLUSION

RGOR grounds audio-driven lip-sync in the dubbed person’s enrollment references and HD mouth patches, and trains against a paired judge. On the reported evaluations, it ranks first or second on every fidelity metric and raters prefer it on every human-study question.

6 CONCLUSION

  • RGOR renders the specific person’s mouth by grounding generation in enrollment references and HD patches and training against a paired judge.The references show the person’s mouth in varied shapes without motion to copy; HD patches carry fine detail past the VAE, and the shape term keeps the opening in step with the real frame.Source: §6 · pp. 9–10
  • RGOR ranks first or second on every fidelity metric and raters prefer it on every human-study question, while preserving the person’s lip and dental detail and leaving the rest of the face untouched.The evaluation protocol withholds the target mouth from systems that accept a separate reference.Source: §6 · p. 10
  • Removing any one of RGOR’s four components degrades the mouth in its own way, and the authors suggest the principle may extend to person-specific hands, eyes, and hair.The components are scattered references, HD patches, a paired judge, and a shape term.Source: §6 · p. 10

A HUMAN STUDY

The human study compares eight anonymized systems on mouth identity, the face outside the mouth, realism, and lip-sync. Oral perceptual and texture metrics correlate positively with raters on every question, unlike pixel-averaging metrics.

A HUMAN STUDY

  • The interface shows a real reference recording above eight anonymized, reshuffled videos; raters choose separately for mouth identity, untouched appearance outside the mouth, realism, and synchronization.All candidates are heard with the same target speech, and raters are told pose and background may differ across takes.Source: A HUMAN STUDY · p. 15
  • Nineteen usable raters judged 40 held-out identities, with each rater assessing a random 20; the study yielded 380 votes per question.Three of 22 participants were removed because their responses were unusable.Source: A HUMAN STUDY · p. 15
  • Oral perceptual and texture metrics correlate positively with human votes on every question, while pixel-averaging metrics do not; LatentSync-1.6 leads on MAE and PSNR but receives few votes.Raters prefer a sharp, detailed mouth to an averaged one. Table 3 reports Spearman rank correlations between vote shares and metrics; its Outside and Realism columns coincide.Source: A HUMAN STUDY · p. 15, Table 3 · p. 15

B PROTOCOL AND METRICS

The protocol evaluates held-out identities using separate enrollment and query clips, with different input conditions for masked inpainting systems and systems that cannot take masked input. It scores mouth and non-mouth regions using pixel, perceptual, texture, opening, and synchronization measures.

B PROTOCOL AND METRICS

  • The benchmark uses 50 held-out NeRSemble identities, each with two query sentences and an enrollment bank of two other sentences; it also includes 48 query clips from 24 HDTF speakers.An enrollment clip is never a query, so the enrollment bank does not contain the target clip’s mouth.Source: B PROTOCOL AND METRICS · p. 15
  • Masked inpainting systems receive masked target frames and enrollment references, while X-Dub and commercial services re-render an enrollment clip with the target audio.MuseTalk-1.5 runs as released with the target frame as its reference; other baselines use officially recommended inference settings.Source: B PROTOCOL AND METRICS · pp. 15, 17
  • Outputs are aligned to the target and scored frame by frame inside a fixed oral box and outside it, using pixel error, LPIPS, DISTS, texture and open-area ratios, and SyncNet synchronization measures.MAE-out measures pixel error outside the oral box; LSE-C and LSE-D use the public SyncNet evaluator.Source: B PROTOCOL AND METRICS · p. 17
  • The texture and open-area ratios are best at one: they compare inner-mouth high-frequency energy and parser-detected mouth-interior area, respectively, with ground truth.Values below or above one indicate smoothing or excess texture for Tex, and too-closed or too-wide openings for Open.Source: B PROTOCOL AND METRICS · p. 17

C SAME-FRAME REFERENCE LEAK

Released inpainting code can use the unmasked target frame as its reference, giving paired evaluations access to the mouth being reconstructed. Tests with a separate same-person reference show lower oral LPIPS and DISTS than the same-frame-reference condition for LatentSync-1.6.

C SAME-FRAME REFERENCE LEAK

  • With the target frame as its reference, LatentSync-1.6’s oral LPIPS is 0.262 versus 0.334 with a different-clip reference, and its DISTS is 0.189 versus 0.234.The released inference code uses the unmasked copy of each edited frame as its reference, so paired reconstruction shows the model the answer.Source: C SAME-FRAME REFERENCE LEAK · p. 17
  • Wav2Lip and X-Dub also move in the same direction with the target clip as input; X-Dub completes 51 of 100 NeRSemble clips in that condition, and MuseTalk-1.5 is run only with the target frame as reference.The X-Dub comparison uses the 51 clips completed in both conditions.Source: C SAME-FRAME REFERENCE LEAK · p. 17
  • Table 4 marks rows whose reference or input is the target clip, enabling comparison of same-frame and different-clip reference conditions.The authors note that some fidelity credited in paired evaluation can therefore be copied from the input.Source: Table 4 · p. 17, C SAME-FRAME REFERENCE LEAK · p. 17

D IMPLEMENTATION DETAILS

Implementation pairs a mouth-specific judge with a shape term, using frozen perception modules to compare rendered mouths with references and preserve mouth geometry. Training updates the parent generator and HD-patch modules.

D IMPLEMENTATION DETAILS

  • The judge compares rendered mouths with references using embeddings from a person-discriminative ResNet-18, while a frozen SegFormer supplies mouth-interior and lip masks for the shape term.The judge uses real mouths of other people in the same batch as negatives.Source: D IMPLEMENTATION DETAILS · p. 17
  • Training updates 771.5M trainable parameters for 1,750 steps on four RTX 5090 GPUs, with peak memory of 21.2 GiB per GPU.Training uses fp16 mixed precision and one 16-frame window per GPU per update.Source: D IMPLEMENTATION DETAILS · p. 17
  • Figure 8 illustrates failures involving fast head turns, low-resolution enrollment, and mouths held too closed or opened too wide.The examples compare ground truth and RGOR at the same frames, alongside the full frame and its fixed oral box.Source: Figure 8 · p. 18

E FAILURE CASES

RGOR has specific boundaries: fixed-box judging mishandles fast turns, reference quality limits retained oral detail, and audio-driven opening amplitude can diverge from a person’s habitual opening. On HDTF, LatentSync-1.6 outperforms RGOR on several oral and synchronization metrics.

E FAILURE CASES

  • When the head turns quickly, RGOR keeps a frontal mouth stretched across the face because judging uses a fixed oral box and misses content that leaves it.The real mouth turns toward the box’s edge.Source: E FAILURE CASES · p. 18
  • Low-resolution enrollment limits the detail RGOR can reproduce; on HDTF, gapped lower teeth become a generic row, and unseen tooth surfaces can likewise revert to a plausible generic mouth.The authors attribute HDTF’s weaker result in part to low-resolution, single-clip enrollment and larger head motion.Source: E FAILURE CASES · p. 18
  • Opening width is predicted from audio and matched to real frames only on average, so RGOR can open too little or too much; its open-area ratio is 0.880 on NeRSemble and 1.39 on HDTF.The authors identify explicit opening-amplitude conditioning, potentially estimated from enrollment clips, as a next step.Source: E FAILURE CASES · p. 18
  • Controlling opening amplitude and judging mouths that move outside the oral box remain future-work directions.Source: E FAILURE CASES · p. 18

E.1 OUT-OF-DISTRIBUTION CHECK ON HDTF

  • On HDTF’s 48-query, 24-speaker strict protocol, LatentSync-1.6 beats RGOR on pixel and perceptual oral metrics and LSE-C and LSE-D.RGOR retains the closest-to-one texture ratio and lowest error outside the oral box, but opens the mouth too wide.Source: §E.1 · p. 19, Table 5 · p. 19

Related papers

Loading 2609.38019v1…