PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling

Jing-Xuan Zhang, Genshun Wan, Jianqing Gao, Jia Pan, Yumei Zhang, Lichen Zhang

arXiv:2610.19559v1cs.CVeess.AS

TL;DR

Lipreading is challenging because visual speech cues are ambiguous and sensitive to pose and resolution. FLARE distills Whisper representations while jointly modeling face and lip inputs, and reports strong results across LRS2, LRS3, and out-of-domain WildVSR.

  • Problem

    Visual speech representation learning is challenged by ambiguous viseme-to-phoneme mappings and sensitivity to head pose and video resolution, while AV-SSL supervision design and face-lip complementarity remain underexplored.

    Source: §1 · pp. 1–2

  • Method

    FLARE uses Whisper audio representations as quantized masked-prediction targets and jointly processes full-face and lip-ROI inputs for visual speech representation learning.

    Source: §1 · p. 2, §3.3 · pp. 4–5

  • Results

    11.7%, 13.2%, and 34.3% WER on LRS2, LRS3, and out-of-domain WildVSR, respectively, are FLARE’s reported best-model results.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FLARE provides a visual speech representation framework that combines cross-modal distillation with face-lip modeling, with design studies confirming contributions from its major components.

    Source: §5 · p. 10

    Joint face-lip modeling adds a visual branch and computational cost, and the study does not assess transfer to broader audio-visual tasks or capture of speaker and other non-linguistic information.

    Source: E Limitations · p. 16

Abstract

from arXiv · show

Lipreading remains challenging due to the inherent ambiguity of visual speech cues and their sensitivity to head pose and video resolution. In this work, we propose FLARE, a framework for robust visual speech representation learning via cross-modal distillation and joint face-lip modeling. FLARE leverages semantically rich Whisper representations as the acoustic teacher signal, converting them into discrete distillation targets through a random-projection quantizer and pretraining the model with a masked prediction objective. To capture complementary visual information, FLARE employs a dual-stream architecture that jointly processes lip region-of-interest (ROI) and full-face inputs, where the lip stream captures finegrained articulatory motion and the face stream encodes broader facial dynamics. During pretraining, auxiliary audio input is incorporated to facilitate cross-modal alignment, and modality dropout is applied to encourage robust visual-only representations. For downstream lipreading, FLARE serves as a visual encoder coupled with either a lightweight Transformer decoder or an LLM-based decoder. Evaluated on LRS2, LRS3, and the out-of-domain WildVSR benchmark, FLARE achieves strong lipreading performance, with word error rates of 11.7%, 13.2%, and 34.3%, respectively. A progressive design study confirms the contribution of each major component. Further analyses demonstrate that joint face-lip modeling improves robustness under degraded video resolution and large head pose variation, while yielding more phoneme-discriminative visual representations. Code and pretrained models are available at https://github.com/jxzhanggg/FLARE.

1 Introduction

Visual speech learning is challenged by ambiguous viseme-to-phoneme mappings and sensitivity to pose and resolution, while prior AV-SSL work leaves supervision design and joint face-lip cues underexplored. FLARE combines Whisper-based cross-modal distillation with face-and-lip modeling and reports strong results on three benchmarks.

1 Introduction

  • Visual speech recognition infers spoken content from lip movements, but ambiguous viseme-to-phoneme mappings and sensitivity to pose and resolution make representation learning challenging.Earlier approaches used visual bottleneck features, followed by end-to-end models trained with text supervision.Source: §1 · p. 1
  • Prior AV-SSL work often learns audio, visual, and joint representations together, while lip-ROI-only inputs can be unreliable under pose variation, low resolution, or imperfect cropping.The introduction identifies face-lip complementarity as insufficiently exploited and understood.Source: §1 · p. 2
  • FLARE distills semantically rich Whisper audio representations into discrete targets for masked prediction and jointly processes full-face and lip-ROI inputs.Auxiliary audio supports cross-modal alignment during pretraining, while modality dropout encourages visual-only representations.Source: §1 · p. 2
  • 11.7%, 13.2%, and 34.3% WER on LRS2, LRS3, and out-of-domain WildVSR, respectively, are FLARE’s reported best-model results.The study also includes a progressive design study and analyses of face-lip complementarity and learned representations.Source: §1 · p. 2

2 Related Works

Related work covers audio-visual cross-modal distillation and the selection of visual inputs for lipreading. FLARE focuses distillation on visual speech representations and integrates face and lip inputs early rather than combining separate models at decision level.

2.1 Audio-Visual Cross-Modal Distillation

  • Audio-visual self-supervised learning uses paired audio-visual speech to learn general representations through objectives such as masked prediction or contrastive learning.Pretrained encoders can be adapted with task-specific decoders, integrated with LLMs, or used for speech-model reprogramming.Source: §2.1 · p. 2
  • FLARE differs from prior cross-modal distillation by using a single large-scale external model and explicitly focusing on visual speech representations rather than jointly optimizing all modalities.Related approaches include teacher-to-lipreading distillation and AV-SSL methods with shared or modality-specific encoders.Source: §2.1 · pp. 2–3

2.2 Visual Input Selection for Lipreading

  • Earlier multi-scale visual approaches mainly trained separate models and combined them at decision level, whereas FLARE jointly models face and lip inputs early.Full-face input can add head-pose and broader facial-dynamics cues to the cropped lip ROI.Source: §2.2 · p. 3

3 Proposed Method

FLARE transfers supervision from Whisper’s audio representations to visual speech and combines lip and face inputs to learn complementary representations. Its design includes multimodal pretraining and two downstream lipreading decoder settings.

3.1 Model Inputs

  • Pretraining uses full-face video, lip-ROI video, and auxiliary audio features; span-based masking corrupts selected input frames.Audio is included to facilitate cross-modal alignment, and its waveform is corrupted with additive noise to prevent over-reliance on acoustic cues.Source: §3.1 · p. 3

3.2 Model Architecture

  • Face and lip video streams are encoded by parallel visual frontends, while a feed-forward layer projects audio features; modality dropout supports training with video-only or missing-face inputs.The remaining features are concatenated and linearly projected before the context encoder processes them for long-range temporal modeling.Source: §3.2 · p. 4
  • An enhanced Conformer context encoder models temporal patterns using feed-forward, convolution, and self-attention modules, with RoPE, grouped-query attention, SwiGLU, and RMSNorm.Sequential context helps reduce ambiguity in viseme-to-phoneme mapping.Source: §3.2 · p. 4

3.3 Cross-Modal Distillation Loss

  • Whisper representations are normalized, projected into a code space, and assigned to nearest random-codebook entries to create discrete targets for masked prediction.Targets are predicted for both masked and unmasked regions, transferring knowledge from the teacher’s deep audio representations.Source: §3.3 · pp. 4–5

3.4 Lipreading Task Finetuning

  • Downstream lipreading pairs the pretrained encoder with either a lightweight Transformer decoder or an LLM-based decoder, used for comparison with prior AV-SSL work and to explore stronger decoding, respectively.The LLM setting uses an adaptor, LoRA, and multistage finetuning to align encoder outputs with the LLM embedding space.Source: §3.4 · p. 5

4 Experiments and Results

FLARE is evaluated against lipreading baselines and through progressive design, robustness, and representation analyses. Results highlight strong benchmark performance and benefits from combining face and lip inputs.

4.1 Experimental Setup

  • Experiments use pretraining corpora totaling 1,759 or 4,966 hours, evaluate on LRS2, LRS3, and WildVSR, and test Transformer and LLM-based decoders.The Base and Large models have about 117M and 342M parameters, respectively; WildVSR uses lip-only inputs because full-face videos are unavailable.Source: §4.1 · p. 5, §4.2 · p. 6

4.2 Comparison with Baselines

  • FLARE reaches 13.2% WER on LRS3, approaching the previous best 12.8% despite that system using 100k hours of training data.The high-resource comparison also reports strong performance on LRS2 and out-of-domain WildVSR; an LLM-based decoder further improves accuracy and surpasses most previously reported results.Source: Table 2 · p. 6, §4.2 · pp. 6–7

4.3 Progressive Design Study

  • Whisper with random-projection quantization performs best or comparably among distillation variants, while replacing semantic audio representations with raw Mel filterbanks substantially reduces accuracy.The study compares Whisper, WavLM/Conformer, and AV-HuBERT teachers, along with regression and target-prediction objectives.Source: §4.3 · p. 7, Table 3 · p. 7

4.4 Robustness to Video Quality and Head Pose

  • Lip-only WER rises with yaw, whereas joint face-lip input performs best across poses and yields clearer phoneme separation; deeper context blocks progressively align visual and audio-visual representations.The yaw analysis reports no statistically significant WER association for face-only or joint inputs.Source: Figure 3 · p. 8, §4.4 · p. 9, Figure 4 · p. 9

4.3 Progressive Design Study

  • Visual-dominant modality dropout (0/0.75/0.25) performs best among the tested dropout settings, and Conformers outperform standard Transformers for context modeling.A modality-scheduling curriculum gives no further gains; transferred dropout benefits the enhanced Conformer with limited finetuning and is comparable with more extensive finetuning.Source: §4.3 · p. 8
  • Input-level span replacement is the most effective tested visual masking strategy, and 3D-ConvNeXt improves on 3D-ResNet after extended pretraining.At 400k steps, the reported WERs are 32.2/27.7% for 3D-ConvNeXt and 33.9/29.5% for 3D-ResNet.Source: §4.3 · p. 8
  • Jointly modeling face and lip inputs substantially reduces recognition errors, consistent with complementary information from broader facial dynamics and fine-grained lip articulation.The robustness analysis finds joint face-lip input has the lowest WER across tested resolution settings.Source: §4.3 · p. 8, Figure 2 · p. 8, §4.4 · p. 9

4.4 Robustness to Video Quality and Head Pose

  • Across downsampled video resolutions, joint face-lip input consistently achieves the lowest WER, while the relative advantage of face-only and lip-only inputs varies by dataset.The tests use 0.75×, 0.5×, and 0.25× of original spatial resolution.Source: Figure 2 · p. 8, §4.4 · p. 9

5 Conclusion

FLARE combines cross-modal distillation with joint face-lip modeling to learn visual speech representations. Experiments report strong lipreading performance and progressive design studies confirm contributions from its major components.

5 Conclusion

  • FLARE combines cross-modal distillation with joint face-lip modeling for visual speech representation learning.Parallel face and lip frontends with lightweight fusion use global facial dynamics and fine-grained articulatory motion.Source: §5 · p. 10
  • Whisper representations provide supervision through random-projection quantization and a masked prediction objective.The conclusion describes this approach as learning effective and robust visual representations for lipreading.Source: §5 · p. 10
  • Experiments report strong lipreading performance, while progressive design studies confirm contributions from each major component.Source: §5 · p. 10
  • The authors present FLARE as a foundation for future research on visual speech understanding, audio-visual speech recognition, and broader multimodal speech tasks.Source: §5 · p. 10

A Lipreading Task Finetuning Details

The finetuning setup adapts the pretrained visual encoder using either a lightweight Transformer decoder or an LLM-based decoder. For the lightweight decoder, training begins with the encoder frozen and then jointly optimizes both components.

A Lipreading Task Finetuning Details

A.1 Lightweight Decoder

  • With the lightweight decoder, only the randomly initialized decoder is trained for the initial k updates before encoder-decoder joint optimization.The encoder first serves as the pretrained model's fixed visual representation.Source: §A.1 · p. 13
  • The lightweight-decoder setup omits auxiliary objectives such as CTC loss to better reflect pretrained visual-representation quality.The setup is also used in prior AV-SSL studies, enabling direct comparison with previous methods.Source: §A.1 · p. 13

A.2 LLM Decoder

The LLM-based decoder uses a staged adaptation process, first training an adaptor with the encoder and LLM frozen, then jointly optimizing the adaptor and LoRA weights. The authors report that this progression stabilizes training and improves final performance over single-stage optimization.

A.2 LLM Decoder

  • LLM finetuning starts from the encoder obtained after joint finetuning with the lightweight Transformer decoder.Source: §A.2 · p. 13
  • The adaptor is first trained alone with the encoder and LLM frozen, then optimized with LLM LoRA weights while other components remain fixed.The adaptor maps visual representations to the LLM embedding space.Source: §A.2 · p. 13
  • The stage-wise design stabilizes training and yields better final performance than single-stage optimization.It progressively aligns the pretrained encoder, adaptor, and decoder.Source: §A.2 · p. 13

B Experimental Setup Details

The experimental setup uses a roughly 4966-hour pretraining corpus, evaluates on LRS2, LRS3, and WildVSR, and specifies model variants and finetuning compute. It also describes table comparisons for visual-input configurations and computational costs.

B Experimental Setup Details

  • Pretraining uses approximately 4966 hours from LRS2, LRS3, VoxCeleb2-English, AVSpeech-English, and AVYT-talking.AVYT-talking contributes approximately 29% of pretraining hours and contains only cropped lip ROIs; the face representation is set to zero for that dataset.Source: B Experimental Setup Details · p. 14
  • Evaluation uses LRS2 and LRS3 test sets plus out-of-domain WildVSR, which is lip-only and evaluated with the face representation set to zero.The low-resource setting uses the 30-hour subset and full LRS3; high-resource pretraining and finetuning use either 1,759 or 4,966 hours.Source: B Experimental Setup Details · p. 14
  • The Base and Large variants have approximately 117M and 342M parameters, respectively, with different hidden sizes and numbers of enhanced Conformer blocks.The Base uses hidden size 512 and 12 blocks; the Large uses hidden size 768 and 18 blocks.Source: B Experimental Setup Details · p. 14
  • Pretraining runs for 600k updates on the 1,759-hour subset and 800k updates on the full 4,966-hour corpus.Lightweight finetuning freezes the encoder for its first 90k of 180k steps; LLM finetuning uses two further stages of 10k and 60k updates.Source: B Experimental Setup Details · p. 14
  • Table 4 reports LRS3-test WER across visual-input configurations for pretraining, finetuning, and testing.Source: Table 4 · p. 15
  • Table 5 compares computational costs across model architectures and visual frontends, measured on a 6-second utterance using one NVIDIA Tesla P40 GPU.Source: Table 5 · p. 15

C Effect of Visual Input Configurations

Joint face-lip inputs offer the strongest LRS3 results, while joint face-lip training with modality dropout also supports flexible single-input testing.

C Effect of Visual Input Configurations

D Computational Cost

FLARE’s additional lip frontend raises computational cost while improving recognition accuracy; the LLM-based decoder adds substantial further cost.

D Computational Cost

  • FLARE and FLARE-LLM costs were compared with DistillAV using 6-second utterances on a single NVIDIA Tesla P40 GPU, with RTF defined as inference time divided by input duration.Lower RTF indicates faster inference.Source: D Computational Cost · p. 15
  • Adding a lip frontend substantially increases computational cost despite a modest increase in model parameters, while bringing recognition-accuracy improvements and keeping Base and Large RTF below 0.24.Source: D Computational Cost · p. 15
  • The 7B-parameter FLARE-LLM has overall RTF values of 1.31 to 1.42, with the LLM accounting for most computation; improving efficiency remains future work.The model uses no token compression.Source: D Computational Cost · p. 16

E Limitations

The paper identifies added model cost and untested transfer beyond lipreading as limitations, discusses potential benefits and risks, and states that its introductory claims match its contributions and scope.

E Limitations

  • Joint face-lip modeling adds a visual branch, increasing model complexity and computational cost relative to lip-only approaches.Transfer to broader audio-visual speech tasks and representation of speaker or other non-linguistic cues are not investigated.Source: E Limitations · p. 16

F Broader Impacts

  • The authors describe potential benefits for assistive communication and multimodal interaction, alongside misuse risks such as covert surveillance and uneven performance across real-world conditions.Potential consequences include incorrect transcriptions, unreliable assistance, and unfair disparities.Source: F Broader Impacts · p. 16

NeurIPS Paper Checklist

  • The checklist states that the abstract and introduction accurately reflect the paper’s contributions and scope, supported by the method and experiments.Source: NeurIPS Paper Checklist · p. 17

2. Limitations

The checklist records limited statistical uncertainty reporting, while documenting experimental settings, reproducibility information, and broader-impact considerations.

2. Limitations

  • The authors state that the architecture, datasets, training pipeline, ablation setup, and implementation choices support reproduction of the main claims.Source: §2 · p. 18
  • The authors state that the paper specifies datasets, train/test settings, model and decoder variants, optimization details, and hyperparameters across the main text and appendix.Source: §2 · p. 19
  • The authors do not report statistical significance or repeated-run variance because repeating large-scale pretraining and finetuning would be computationally expensive.The main benchmark tables report WER only, following common practice in prior work.Source: §2 · p. 19
  • The checklist reports that hardware and corresponding training time are provided in Appendix B.Source: §2 · p. 20

10. Broader impacts

  • The authors state that they comply with the NeurIPS Code of Ethics and adhere to the usage restrictions of existing assets, citing sources and licenses in Appendix B.Source: §2 · p. 20, §10 · p. 21
  • The checklist states that the paper discusses potential positive and negative societal impacts in Appendix F.Source: §10 · p. 20
  • The authors do not consider the work to present unusually high misuse risks requiring dedicated technical safeguards, while intending future releases for legitimate research subject to licenses and privacy considerations.Source: §10 · p. 21
Loading 2610.19559v1…