PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Practical Lipreading with Distilled and Efficient Models

Pingchuan Ma, Brais Martinez, Stavros Petridis, Maja Pantic

arXiv:2007.06504v3cs.CVcs.LG

TL;DR

Lipreading research still faces a gap between computationally costly full models and efficient models that trail them in accuracy, limiting practical deployment. This paper combines self-distillation, lightweight architectures, and knowledge distillation for compact models. It reports state-of-the-art results on LRW and LRW-1000, and an LRW model matching state-of-the-art with substantially lower compute and parameter counts.

  • Problem

    Full lipreading models are computationally costly, while complexity-focused models trail them in accuracy; practical on-device use also depends on memory and battery consumption.

    Source: §1 · p. 1

  • Method

    The paper uses born-again distillation to improve the state-of-the-art model and designs lightweight backbones and a DS-TCN head, training efficient models through knowledge distillation.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    8.2× fewer FLOPs and 3.9× fewer parameters let the distilled ShuffleNet v2 (1×)+DS-MS-TCN match the previous LRW state-of-the-art accuracy; the paper also reports new state-of-the-art performance on LRW and LRW-1000.

    Source: §1 · p. 2, §4 · pp. 3–4

  • Takeaways & Limitations

    The paper presents efficient visual speech recognition models with performance similar to current state-of-the-art while reducing computational cost by about 8 times.

    Source: §5 · p. 4

Abstract

from arXiv · show

Lipreading has witnessed a lot of progress due to the resurgence of neural networks. Recent works have placed emphasis on aspects such as improving performance by finding the optimal architecture or improving generalization. However, there is still a significant gap between the current methodologies and the requirements for an effective deployment of lipreading in practical scenarios. In this work, we propose a series of innovations that significantly bridge that gap: first, we raise the state-of-the-art performance by a wide margin on LRW and LRW-1000 to 88.5% and 46.6%, respectively using self-distillation. Secondly, we propose a series of architectural changes, including a novel Depthwise Separable Temporal Convolutional Network (DS-TCN) head, that slashes the computational cost to a fraction of the (already quite efficient) original model. Thirdly, we show that knowledge distillation is a very effective tool for recovering performance of the lightweight models. This results in a range of models with different accuracy-efficiency trade-offs. However, our most promising lightweight models are on par with the current state-of-the-art while showing a reduction of 8.2x and 3.9x in terms of computational cost and number of parameters, respectively, which we hope will enable the deployment of lipreading models in practical applications.

1. INTRODUCTION

Lipreading models face a practical trade-off: full models are computationally costly, while efficient models have lagged in accuracy. The paper combines self-distillation and lightweight architecture design to improve accuracy and recover performance in efficient models.

1. INTRODUCTION

  • Practical deployment is constrained by model computation, memory footprint, and battery consumption, while existing complexity-focused models trail full-fledged models in accuracy..Source: §1 · p. 1
  • Born-again distillation iteratively trains same-architecture student-teacher pairs, with each student becoming the next generation’s teacher until improvement stops.The paper also proposes intermediate-capacity teachers to bridge the architectural gap between full-fledged and efficient networks because using only the best-performing model is suboptimal.Source: §1 · p. 1
  • The proposed DS-TCN head replaces standard temporal convolutions with depthwise separable convolutions to reduce parameters and FLOPs.The paper identifies the classifier head as lacking an efficient equivalent to the lightweight backbones used in prior designs.Source: §1 · p. 1
  • The method reports new state-of-the-art performance on LRW and LRW-1000 by a wide margin without increased computational performance, and matches LRW state-of-the-art using 8.2× fewer FLOPs and 3.9× fewer parameters.Source: §1 · p. 2

2. TOWARDS PRACTICAL LIPREADING

The paper builds on a ResNet-18 and multi-scale TCN lipreading model, then makes the backbone and temporal head more efficient. It trains models with hard-label and teacher-output distillation losses.

2. TOWARDS PRACTICAL LIPREADING

  • The base architecture combines a modified ResNet-18 backbone with a multi-scale temporal convolutional network for short- and long-term information.Its first convolution is a 3D convolution with a 5 × 7 × 7 kernel, followed by the standard design through global average pooling.Source: §2 · p. 2
  • ShuffleNet v2 replaces ResNet-18 as the efficient spatial backbone, while DS-TCN replaces standard MS-TCN convolutions with depthwise separable ones.ShuffleNet v2 (1.0×) has 5× fewer parameters and 12× fewer FLOPs than ResNet-18; DS-TCN reduces convolution cost from k × Cin × Cout to k × Cin + CinCout.Source: §2 · p. 2, Figure 2 · p. 2
  • Born-again distillation uses the student from one generation as the teacher for the next, and an ensemble may combine predictions across generations.The table caption specifies that each student is trained using the model on the preceding line as teacher.Source: §2 · p. 2, Table 1 · p. 3
  • The training objective combines cross-entropy for hard labels with KL-divergence distillation loss for soft student and teacher predictions.The paper uses born-again distillation for the state-of-the-art model and standard knowledge distillation for a series of smaller-capacity models.Source: §2 · p. 2

3. EXPERIMENTAL SETUP

Experiments use LRW and LRW-1000, with different dataset preparation, and describe the training protocol and augmentation used for the models.

Datasets

  • LRW contains over half a million utterances of 500 English words from over 1,000 speakers; LRW-1000 has 718,018 utterances across 1,000 Mandarin syllable classes.LRW utterances have 29 frames; LRW-1000 utterances vary from 0.01 to 2.25 seconds.Source: §3 · p. 3

Pre-processing

  • LRW mouth regions are cropped to 96×96 after 68 facial landmarks are detected and tracked; LRW-1000 videos are already cropped.Source: §3 · p. 3

Training

  • Models use AdamW with weight decay 0.01 and train for 80 epochs from random initialization, without external datasets.The initial learning rate is 0.0003, batch size is 32, and learning-rate decay uses cosine annealing without warm-up.Source: §3 · p. 3

Data Augmentation

  • Training augmentation includes horizontal flipping, random 88×88 crops, mixup at weight 0.4, and variable-length augmentation.Testing uses the 88×88 center patch.Source: §3 · p. 3

Evaluation Scope

  • Table 2 compares LRW-1000 classification accuracy on the publicly available cropped-mouth data; each student uses the preceding row's model as teacher.The caption distinguishes a full-face result that uses an unavailable database version.Source: Table 2 · p. 3

4. RESULTS

Self-distillation improves benchmark results, while lightweight architectures and sequential distillation produce accuracy–efficiency trade-offs. The distilled models match LRW state-of-the-art accuracy with substantially lower computation and parameter counts.

Born-Again Distillation

  • On LRW, the strategy beat the previous state-of-the-art by a 2.6% margin without added computational cost, and an ensemble reached 88.5% accuracy; LRW-1000 single-model and ensemble gains were 3.9% and 5.2%.The LRW-1000 comparisons were limited to works using the database’s publicly available version.Source: §4 · p. 3

Efficient Lipreading

  • On LRW, replacing MS-TCN with DS-MS-TCN cut parameters to almost one third and FLOPs by 50% while maintaining ShuffleNet v2 accuracy.A single TCN head and 0.5× ShuffleNet v2 width reduced accuracy by 3.4% and a further 1.9%, respectively, to 78.1%.Source: §4 · pp. 3–4, Table 3 · p. 4
  • On LRW-1000, ShuffleNet v2 with a single TCN head dropped 0.7% versus the full model; DS-TCN dropped a further 1.6%, while 0.5× width matched baseline ShuffleNet v2 accuracy.Source: §4 · p. 4, Table 4 · p. 4

Sequential Distillation

  • Sequential distillation uses intermediate-capacity networks as teachers for lower-capacity students, progressively bridging the full model's architecture to the target.The standard approach of using the best-performing model directly as teacher was also explored; the authors note that a wider architectural gap may make transfer less effective.Source: §4 · p. 4
  • On LRW, ShuffleNet v2 (1×)+DS-MS-TCN matched in accuracy with 8.2× fewer FLOPs and 3.9× fewer parameters; on LRW-1000 it dropped 1.2% with 18.8× fewer FLOPs and 22.9× fewer parameters.Source: §4 · p. 4, Table 3 · p. 4, Table 4 · p. 4

5. CONCLUSIONS

Knowledge distillation yields state-of-the-art results on isolated-word recognition, while efficient visual-speech models achieve results similar to current state-of-the-art at lower computational cost.

5. CONCLUSIONS

  • Knowledge distillation yields state-of-the-art results on isolated-word recognition.The conclusion identifies knowledge distillation as the approach.Source: §5 · p. 4
  • Efficient visual-speech models achieve results similar to current state-of-the-art while reducing computational cost by 8 times.The conclusion reports this efficiency-performance trade-off for the investigated models.Source: §5 · p. 4
  • The authors identify cross-modal distillation for audiovisual speech recognition as a future research direction.The conclusion says its effect on audiovisual model performance would be interesting to investigate.Source: §5 · p. 4
Loading 2007.06504v3…