PaperlayerView exact revision on arXiv

Source-linked AI summary

Lip-Focused Audio-Visual Consistency Detects Speech Content Manipulation beyond Coarse Synchronization

Jian Zong, Chenghan Lin, Meng Liu, Zhao Xu, Junyu Wang, Siyuan Zhang, Peiyuan Jiang, Longbiao Wang, Jianwu Dang

arXiv:2610.30872v1eess.AS

TL;DR

Speech-content manipulation can change what a visible person appears to say while preserving identity and, with retalking, coarse synchrony. C-MIND tests local audio–lip consistency using aligned frame-level representations and explicit agreement/disagreement interactions. Pretrained SyncLip-C performs strongly on SyncSpeech, but the reported Video-Retalking result shows generator sensitivity.

  • Problem

    Speech replacement can evade texture, identity, and coarse synchronization cues, motivating tests of whether local audio–lip compatibility remains informative after those cues are controlled.

    Source: §1 · p. 2

  • Method

    C-MIND aligns speech and mouth-motion representations at frame level and models their agreement and disagreement without transcripts, ASR features, or an unmanipulated reference.

    Source: §1 · p. 2, §3.1 · p. 4, §3.3 · p. 5

  • Results

    95.813% scenario-averaged accuracy and 0.9943 AUC for pretrained SyncLip-C on SyncSpeech contrast with 82.775% accuracy on unseen Video-Retalking attacks.

    Source: §6 · p. 13, §5.5 · p. 13

  • Takeaways & Limitations

    Within the evaluated settings, lip-focused speech-articulation consistency is a useful forensic signal when texture artifacts, identity mismatch, and coarse synchronization are weakened as shortcuts.

    Source: §6 · p. 13

    Accuracy falls to 82.775% on Video-Retalking scenarios 5–6, and the study lacks compression, codec, resolution, and audio-noise stress tests.

    Source: §5.5 · p. 13

Abstract

from arXiv · show

Speech-content manipulation changes what a visible person appears to say by replacing the speech track and, in stronger attacks, regenerating mouth motion to match the forged utterance. Such attacks are difficult to expose with texture artifacts or coarse lip-sync scores because modern retalking models explicitly optimize audio-visual synchrony. We test the narrower hypothesis that speech replacement still perturbs the local relationship between acoustic dynamics and mouth motion after speaker identity or coarse synchrony is controlled. To isolate this signal, we introduce SyncSpeech, an LRS3-derived benchmark containing real videos, audio-splicing cheapfakes and Wav2Lip retalking attacks under same-speaker and different-speaker replacement. We then introduce C-MIND, a lip-focused detector that aligns audio and mouth-motion representations at frame level and models their agreement and disagreement without transcripts, automatic speech recognition features or an unmanipulated reference. On SyncSpeech, pretrained SyncLip-C achieves 95.813% scenario-averaged accuracy and 0.9943 AUC, outperforming sentence-level fusion, full-face inputs and a reproduced SyncNet baseline under the reported protocol. When trained on the corresponding FakeAVCeleb training split, C-MIND correctly classifies 278 of 280 samples in the selected unseen-speaker test subset, whereas accuracy falls to 82.775% on unseen Video-Retalking attacks. These results identify lip-focused speech-articulation consistency as a useful but generator-sensitive forensic signal within the evaluated settings.

1 Introduction

Speech-content manipulation changes a visible speaker’s message while preserving their apparent identity, and stronger retalking attacks can also restore coarse audio-visual synchrony. The paper tests whether frame-level audio–lip compatibility can expose these replacements despite those controls.

1 Introduction

  • Speech synthesis and visual editing tools support production and assistive uses but can also make public figures, financial authorities, or enrolled users appear to say things they never said.The paper situates its forensic question amid plausible speech synthesis, voice conversion, visual synthesis, and lip-synchronization systems.Source: §1 · pp. 1–2
  • Speech-content manipulation changes the meaning of a message without changing who appears in the video, and attacks may splice audio or regenerate mouth motion to match it.Texture artifacts, identity checks, and coarse lip-sync offsets can be weak cues in this setting.Source: §1 · p. 2
  • Frame-level audio–lip compatibility may reveal local coupling disturbances that persist despite coarse synchronization, without requiring transcript-level semantic verification.Whole-face features may dilute articulation cues, while sentence-level pooling may suppress short inconsistencies.Source: §1 · p. 2
  • The paper introduces SyncSpeech to separate content replacement from speaker-identity mismatch across real, audio-splicing, and retalking scenarios.It evaluates seen and unseen speakers, one unseen retalking generator, and within-dataset unseen-speaker generalization on a selected FakeAVCeleb subset.Source: §1 · p. 2
  • Pretrained SyncLip-C reaches 95.813% mean accuracy and 0.9943 AUC on SyncSpeech, while the lower Video-Retalking result bounds the evidence to articulation consistency rather than general deepfake detection.C-MIND compares aligned speech and mouth-motion embeddings through explicit frame-level interaction.Source: §1 · p. 2

2 Related work and positioning

Prior deepfake detectors use artifact, identity, synchronization, or content-consistency cues, each with assumptions that may limit its use for speech-content manipulation. C-MIND instead examines aligned mouth-focused interactions, while the paper notes that generator and dataset artifacts remain unresolved concerns.

2 Related work and positioning

  • Artifact, identity, and claim-conditioned methods rely on generator traces, identity evidence, or asserted facts, while word-conditioned facial-motion analysis requires person-specific linguistic annotations [20].These cues address important forgery signals but are not assumed by this work.Source: §2 · pp. 2–3
  • Synchronization methods use coarse alignment, reconstructed lip sequences [21], or self-supervised lip and full-face features, but retalking can repair coarse lip sync.Reconstruction quality or coarse alignment can therefore limit sensitivity to retalking attacks.Source: §2 · p. 3
  • Fine-grained content-consistency methods model phoneme-conditioned lip shapes [15], phoneme-viseme correlations [5], or independently recovered transcripts, while some require person-specific word-motion evidence.C-MIND targets content consistency without phoneme labels, ASR transcripts, visual speech recognition, or an enrolled person.Source: §2 · p. 3
  • Self-supervised correspondence detectors learn normal audio-visual structure or representations from real videos, whereas C-MIND isolates mouth-focused aligned interactions after retalking repairs coarse synchrony.The cited approaches include real-only autoregressive anomaly detection, masked cross-modal learning, and synchronization frameworks.Source: §2 · p. 3
  • Same-speaker and audio-only versus lip-regenerated controls reduce obvious identity and synchronization shortcuts, but cannot eliminate all generator or dataset artifacts.The paper identifies within-benchmark ablations as its strongest evidence and calls for broader comparison and additional unseen generators.Source: §2 · p. 3

3 Method

C-MIND treats speech-content manipulation as a local audio–lip consistency problem: it encodes synchronized audio and mouth motion, aligns their representations, and classifies their frame-level interactions. Its variants contrast sentence-level fusion, frame-level fusion, explicit interaction, and full-face inputs.

3.1 Problem formulation and overview

  • C-MIND predicts whether a synchronized audio and mouth-motion pair is real or manipulated without transcripts, ASR-derived semantic features, or an unmanipulated reference.The design targets local acoustic-articulatory perturbations that may remain after retalking repairs coarse lip synchrony.Source: §3.1 · pp. 3–4
  • Separate encoders produce temporal representations that are projected to a common dimension and aligned frame by frame before SyncLip-C combines concatenation, element-wise product, and absolute difference.The figure depicts the main SyncLip-C pipeline; the controlled SyncLip-A, SyncLip-B, and SyncLip-C variants are not all shown.Source: §3.1 · p. 4, Figure 1 · p. 4

3.2 Audio-lip encoding and temporal alignment

  • A learned audio projection and visual interpolation map the branches to a shared sequence length and dimension, preserving short inconsistencies through frame-level alignment.The visual branch focuses on the mouth region to reduce variation from pose, identity texture, and expression outside the articulators.Source: §3.2 · p. 4

3.3 Frame-level cross-modal interaction

  • At each aligned time step, SyncLip-C combines modality-specific embeddings, their element-wise product for agreement, and their absolute difference for disagreement.Adaptive statistics pooling converts the variable-length sequence to a fixed-dimensional representation, and an MLP produces the manipulation logit.Source: §3.3 · p. 5
  • The decoder variants represent sentence-level fusion, frame-level concatenation, and frame-level explicit interaction, with SyncFace variants substituting aligned full-face input for mouth-region input.The detector outputs the probability that the audio-visual pair is manipulated.Source: §3.3 · p. 5

3.4 Training objective

  • The detector is optimized with binary cross-entropy, and the main pretrained configuration initializes both encoders from a separately trained audio-visual lip-motion speaker-verification model.Projection, interaction, and classification layers are newly initialized in that configuration.Source: §3.4 · p. 5

4 Experiments

The experiments evaluate speech-content manipulation across SyncSpeech scenarios that separate speaker identity from attack type. They describe benchmark construction, preprocessing, model comparisons, and a descriptive human-observer protocol.

4.1 Datasets and evaluation scenarios

  • SyncSpeech evaluates real videos, audio-splicing attacks, and Wav2Lip attacks across same-speaker and different-speaker replacement, with Video-Retalking scenarios reserved for unseen-generator evaluation.The homologous attacks keep audio identity consistent with the target speaker; the benchmark is derived from LRS3.Source: §4.1 · pp. 5–6, Figure 2 · p. 6
  • The test set contains 320 real and 1,280 fake videos, while the unseen-generator Video-Retalking set contains 627 generated fakes; FakeAVCeleb evaluation uses a selected 280-sample unseen-speaker subset.FakeAVCeleb is evaluated after training on its corresponding training split, so it measures within-dataset unseen-speaker generalization rather than zero-shot cross-dataset transfer.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 Manipulation generation and split controls

  • Replacement speech was paired with target videos under same-speaker or different-speaker conditions, and failed face-tracking or synthesis outputs were excluded from generated subsets.Gender was balanced during pairing, and train/test splitting followed the seen- or unseen-speaker setting.Source: §4.2 · p. 7

4.3 Implementation details

  • Examples use synchronized 2-second clips with 96×96 grayscale mouth crops and 80-bin log-mel audio features; the main detector configuration freezes the pretrained backbone.Audio uses 16 kHz sampling, a 25 ms analysis window, and a 10 ms hop; pretraining used MFCC inputs, so the detector's acoustic projection was newly initialized.Source: §4.3 · pp. 7–8

4.4 Baselines, metrics and model selection

  • Comparisons test sentence-level fusion, frame-level concatenation, explicit frame interaction, lip versus full-face inputs, pretraining, and a reproduced SyncNet baseline.Checkpoints are selected by validation AUC, and the validation-balanced-accuracy threshold is fixed before test evaluation; FakeAVCeleb results are treated as within-dataset evidence.Source: §4.4 · p. 8

4.5 Human observer protocol

  • Ten gender-balanced, highly English-proficient participants aged 21–27 judged videos by scenario, providing descriptive context rather than model-selection or hypothesis-testing evidence.Responses are reported only in aggregate scenario-level form.Source: §4.5 · p. 8

5 Results and Discussion

Across SyncSpeech ablations, generalization tests, and interpretation, frame-level lip-focused comparison is the strongest tested signal, while unseen-generator performance remains limited. The results support articulation-level consistency within the evaluated settings, not broad robustness guarantees.

5.1 Ablation validates lip-focused frame-level interaction

  • 95.813% mean accuracy and 0.9943 AUC for pretrained SyncLip-C on SyncSpeech; frame-level lip models outperform sentence-level and full-face variants under the reported protocol.Sentence-level full-face fusion reaches 31.688% mean accuracy and 0.5075 AUC; scratch SyncLip-C reaches 94.313% mean accuracy and 0.9872 AUC.Source: §5.1 · p. 9, Table 2 · p. 9

5.2 Generalization across speakers, datasets and generators

  • Seen/unseen speaker accuracy is 97.250%/94.375%; C-MIND classifies 278 of 280 selected FakeAVCeleb unseen-speaker samples, but reaches 82.775% on unseen Video-Retalking attacks.The FakeAVCeleb result uses its corresponding training split; Video-Retalking performance is only moderately above SyncNet’s 80.223%.Source: §5.2 · pp. 9–11, Table 3 · p. 10, Table 4 · p. 11

5.3 Exploratory human-observer comparison

  • Human observers averaged 67.750% accuracy, including 35.00% on homologous Wav2Lip attacks; this reader study is descriptive, not a model-selection or hypothesis-testing result.Pretrained SyncLip-C reaches 95.813% mean accuracy on SyncSpeech, but the paper treats the gap as qualitative motivation rather than independent proof of superiority over human judgment.Source: §5.3 · pp. 11–12, Table 5 · p. 12

5.4 Uncertainty and scope of the reported results

  • The reported estimates are fixed-test-set results, with 95% Wilson intervals of 94.72–96.69% on 1,600 SyncSpeech videos and 79.62–85.53% on 627 Video-Retalking videos.These intervals quantify sampling uncertainty, but do not replace per-seed dispersion, bootstrap AUC intervals, or compression and audio-noise stress tests.Source: §5.4 · p. 12

5.5 Interpretation, limitations and scope

  • Within SyncSpeech, frame-level speech–mouth comparison is more informative than tested full-face or sentence-level alternatives, supporting local articulation compatibility over generic appearance or global matching.C-MIND uses no transcripts, ASR features, or unmanipulated reference; decisions may combine co-articulation, speaker-specific motion, and generator-dependent mouth dynamics.Source: §5.5 · p. 12
  • Generator shift remains a clear boundary: accuracy falls to 82.775% on Video-Retalking scenarios 5–6, and compression, codec, resolution, and audio-noise stress tests are absent.Strong selected-subset FakeAVCeleb performance is supporting external-benchmark evidence, not a zero-shot cross-dataset claim.Source: §5.5 · p. 13

6 Conclusion

SyncSpeech and C-MIND frame speech-content manipulation as a local audio–lip consistency problem; pretrained SyncLip-C reaches 95.813% scenario-averaged accuracy and 0.9943 AUC on SyncSpeech. The signal remains generator-sensitive, with 82.775% accuracy on unseen Video-Retalking attacks and no evaluation of several real-world degradation shifts.

6 Conclusion

  • 95.813% scenario-averaged accuracy and 0.9943 AUC on SyncSpeech support articulation-level consistency as a useful forensic signal under weakened texture, identity, and synchrony shortcuts.The conclusion reports that pretrained SyncLip-C achieves these results.Source: §6 · p. 13
  • SyncSpeech separates audio-only from lip-regenerated attacks and same-speaker from different-speaker replacement, while C-MIND compares aligned speech and mouth-motion representations.The approach targets local audio–lip consistency rather than whole-face appearance or sentence-level fusion.Source: §6 · p. 13
  • 82.775% accuracy on unseen Video-Retalking attacks shows that the signal is not yet generator-invariant.The evaluation also omits additional generators, compression, codec changes, resolution shifts, and audio noise.Source: §6 · p. 13
Loading 2610.30872v1…