PaperlayerView exact revision on arXiv

Source-linked AI summary

End-To-End Visual Speech Recognition With LSTMs

Stavros Petridis, Zuwei Li, Maja Pantic

arXiv:1701.05847v1cs.CVcs.CL

TL;DR

Visual speech offers information complementary to audio, but visual speech recognition has seen limited joint learning of feature extraction and classification. This paper uses an end-to-end, two-stream LSTM system and reports state-of-the-art performance on OuluVS2 and CUAVE against models with a similar visual front end.

  • Problem

    Visual speech recognition traditionally separates feature extraction from classification, and research jointly learning both stages has been limited.

    Source: §1 · p. 1

  • Method

    A two-stream end-to-end system extracts features from raw mouth and difference images, models each stream with an LSTM, and fuses them using a BLSTM.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    The system is reported to achieve state-of-the-art performance on OuluVS2 and CUAVE against models with a similar visual front end, including a 1.5% improvement on CUAVE.

    Source: §1 · p. 2, §6 · p. 4

  • Takeaways & Limitations

    The authors state that the model can be extended to multiple streams and plan to add an audio stream for audiovisual speech recognition.

    Source: §6 · p. 4

Abstract

from arXiv · show

Traditional visual speech recognition systems consist of two stages, feature extraction and classification. Recently, several deep learning approaches have been presented which automatically extract features from the mouth images and aim to replace the feature extraction stage. However, research on joint learning of features and classification is very limited. In this work, we present an end-to-end visual speech recognition system based on Long-Short Memory (LSTM) networks. To the best of our knowledge, this is the first model which simultaneously learns to extract features directly from the pixels and perform classification and also achieves state-of-the-art performance in visual speech classification. The model consists of two streams which extract features directly from the mouth and difference images, respectively. The temporal dynamics in each stream are modelled by an LSTM and the fusion of the two streams takes place via a Bidirectional LSTM (BLSTM). An absolute improvement of 9.7% over the base line is reported on the OuluVS2 database, and 1.5% on the CUAVE database when compared with other methods which use a similar visual front-end.

1. INTRODUCTION

Visual speech provides information complementary to audio and is unaffected by acoustic noise, while prior visual speech systems generally separated feature extraction from classification. The paper presents an end-to-end, two-stream LSTM system and reports improvements on OuluVS2 and CUAVE.

1. INTRODUCTION

  • Visual speech carries complementary and redundant information alongside audio and is not affected by acoustic noise, potentially improving recognition in noisy environments.Speech comprises audio vocalisation and the corresponding mouth configuration.Source: §1 · p. 1
  • Traditional visual speech recognition separates mouth-ROI feature extraction from classification, with DCT a common feature method.Deep approaches also commonly use bottleneck features followed by a separate classifier.Source: §1 · p. 1
  • End-to-end visual speech recognition was limited, and the cited earlier lipreading system did not outperform state-of-the-art results.The prior system used raw mouth ROIs with LSTM layers and was evaluated on GRID in a subject-dependent experiment.Source: §1 · pp. 1–2
  • The system jointly learns feature extraction and classification from raw mouth and difference images, using LSTMs per stream and a BLSTM to fuse them.The raw-mouth stream encodes static information, while the difference-image stream encodes local temporal dynamics.Source: §1 · p. 2
  • Subject-independent experiments on OuluVS2 and CUAVE report an absolute improvement of 9.7% over the baseline on OuluVS2, and 1.5% on CUAVE compared with methods using a similar visual front-end.Source: §1 · p. 2

2. DATABASES

The study evaluates visual speech recognition on OuluVS2 and CUAVE, which differ in speakers, utterance types, and mouth-ROI preparation.

2. DATABASES

  • OuluVS2 provides 52 speakers and 10 utterances spoken three times each, yielding 156 examples per utterance.The provided mouth ROIs are downscaled to 26 by 44 pixels while keeping the aspect ratio constant.Source: §2 · p. 2
  • CUAVE provides 36 subjects speaking digits 0–9 five times each, yielding 180 examples per digit.The study uses the normal, frontal-position portion of the database.Source: §2 · p. 2
  • For CUAVE, tracked facial points and a neutral reference frame are used to align faces before extracting 90-by-150 mouth crops and downscaling them to 30 by 50 pixels.The crops are centered using tracked points after an affine alignment based on five stable facial points.Source: §2 · p. 2

3. END-TO-END VISUAL SPEECH RECOGNITION

The system extracts features from raw mouth and difference images in separate streams, models temporal dynamics with LSTMs, and fuses the streams with a BLSTM for frame-level labels.

3. END-TO-END VISUAL SPEECH RECOGNITION

  • The raw-mouth stream primarily encodes static information, while the difference-image stream captures local temporal dynamics between consecutive frames.Both streams extract features directly from raw inputs and use bottleneck architectures to compress the images.Source: §3 · p. 2
  • Each stream uses a 2000-, 1000-, and 500-unit sigmoid encoding stack followed by a linear bottleneck, with layers pretrained greedily using RBMs.First- and second-derivative features are computed from the bottleneck features and appended to the bottleneck layer.Source: §3 · pp. 2–3
  • An LSTM models temporal dynamics in each stream, and concatenated LSTM outputs feed a BLSTM that fuses information and produces a softmax label for each frame.The complete system is trained end-to-end so the encoding layers learn features useful for LSTM classification.Source: §3 · p. 3

4. EXPERIMENTAL SETUP

The experiments use subject-independent train, validation, and test splits on both datasets, with utterance-level mean subtraction and image z-normalization before training.

4.1. Evaluation Protocol

  • OuluVS2 uses 40 subjects for training and validation and 12 for testing, with the 40 split into 30 training and 10 validation subjects; CUAVE uses odd-numbered subjects for testing and divides even-numbered subjects into training and validation.These splits yield 900/300/360 OuluVS2 and 600/300/900 CUAVE training, validation, and test utterances, respectively.Source: §4.1 · p. 3

4.2. Preprocessing

  • To reduce subject-dependent characteristics, each frame has its utterance-mean image subtracted, and each image is z-normalized before RBM pretraining.The specified z-normalization sets the mean and standard deviation to 0 and 1, respectively.Source: §4.2 · p. 3

4.3. Training

  • RBMs are pretrained for 20 epochs, then the full system is trained with AdaDelta, mini-batches of 20 utterances, early stopping, and LSTM gradient clipping.The label of the last frame in each utterance is used to label the entire utterance.Source: §4.3 · p. 3
  • The CUAVE confusion matrix labels both axes with digits 0–9.The figure provides a digit-by-digit view of classification confusions.Source: Figure 3 · p. 4

5. RESULTS

The end-to-end two-stream model achieves 84.5% accuracy on OuluVS2, above the 74.8% HMM-DCT baseline despite using a more challenging subject split. On CUAVE, it outperforms methods with a similar visual front-end; confusion patterns and encoder experiments provide additional results.

5. RESULTS

  • 84.5% accuracy for the OuluVS2 end-to-end 2-stream model exceeds the 74.8% HMM-DCT baseline despite using 40 subjects for training and validation and 12 for testing, versus 51 and one.The baseline used leave-one-subject-out cross-validation.Source: §5 · p. 4
  • On CUAVE, the 2-stream model outperforms all approaches with a similar visual front-end, while only achieves higher performance using a more complicated AAM front-end and 6-fold cross-validation.The study used 18 subjects for training and validation and 18 for testing; the database has no standard evaluation protocol, and only and use the same protocol.Source: §5 · p. 4
  • OuluVS2 most frequently confuses Hello with Thank you and See you with Have a good time; CUAVE most frequently confuses number pairs zero/two and six/nine.The CUAVE pairs share similar viseme sequences near the utterance end and start, respectively.Source: §5 · p. 4, Figure 4 · p. 4
  • Convolutional neural network encoding layers performed worse than the proposed system, while data augmentation improved performance but did not exceed it.The authors suggest the CNN result may be due to small training sets and note that also reported it.Source: §5 · p. 4

6. CONCLUSION

The paper presents an LSTM-based end-to-end visual speech recognition system that jointly learns pixel-level feature extraction and classification. Subject-independent experiments show state-of-the-art performance on OuluVS2 and CUAVE against models with a similar visual front end; adding an audio stream is a planned extension.

6. CONCLUSION

  • The system jointly learns to extract features directly from pixels and perform classification using LSTM networks.The conclusion describes it as an end-to-end visual speech recognition system.Source: §6 · p. 4
  • Subject-independent experiments show state-of-the-art performance on OuluVS2 and CUAVE compared with models using a similar visual front end.Source: §6 · p. 4
  • The authors plan to add an audio stream to the multiple-stream model and evaluate it on audiovisual speech recognition tasks.Source: §6 · p. 4
Loading 1701.05847v1…