Source-linked AI summary
Vid2speech: Speech Reconstruction from Silent Video
Ariel Ephrat, Shmuel Peleg
TL;DR
Speechreading is difficult because several phonemes can correspond to one visible speech unit, and producing intelligible audio from silent video also requires an estimable, synthesizable acoustic representation. This paper uses a CNN to predict sound features from temporal face-video clips and synthesizes speech from them. The authors report improved reconstructed-speech intelligibility and promising out-of-vocabulary reconstruction, while identifying less constrained video and speaker-independent or multi-speaker reconstruction as future directions.
Problem
Method
A CNN learns visual features from raw face video and predicts audio feature vectors for each frame using neighboring frames as temporal context, after which the features are synthesized into a waveform.
Source: §1 · p. 2, §3.1 · p. 2, §3.2 · p. 3Results
Takeaways & Limitations
The work demonstrates the feasibility of reconstructing intelligible speech from silent video and shows that regression-based out-of-vocabulary reconstruction holds promise.
Source: §5 · p. 4Further work is needed to establish viability on less constrained video and to generalize reconstruction to speaker-independent and multiple-speaker settings.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Speechreading is a notoriously difficult task for humans to perform. In this paper we present an end-to-end model based on a convolutional neural network (CNN) for generating an intelligible acoustic speech signal from silent video frames of a speaking person. The proposed CNN generates sound features for each frame based on its neighboring frames. Waveforms are then synthesized from the learned speech features to produce intelligible speech. We show that by leveraging the automatic feature learning capabilities of a CNN, we can obtain state-of-the-art word intelligibility on the GRID dataset, and show promising results for learning out-of-vocabulary (OOV) words.
1. INTRODUCTION
Speechreading is difficult because several phonemes can correspond to one visible speech unit, and this paper presents a CNN that predicts speech features from silent video. The authors frame regression as a route to intelligible reconstruction, including words outside the training vocabulary.
1. INTRODUCTION
- Speechreading is difficult because several phonemes can correspond to a single viseme, making reliable phonetic information hard to obtain from a speaker’s face.Source: §1 · p. 1
- Possible applications include videoconferencing in noisy environments, conversations amid loud music, and listening through surveillance video.Source: §1 · p. 1
- The paper models speechreading as articulatory-to-acoustic mapping, using an audio feature vector as the label for each short video segment.The introduction contrasts this with classification approaches based on predefined word, phoneme, or viseme segments.Source: §1 · p. 1
- Regression does not depend on word or sub-word segmentation or manually annotated labels, and can reconstruct words absent from training when phonemes and visemes are sufficiently represented.The authors also describe the approach as vocabulary-agnostic and note its potential to reconstruct non-textual speech such as emotion and prosody.Source: §1 · pp. 1–2
- The CNN takes raw visual data and automatically learns features for acoustic reconstruction, bypassing the need to hand-craft visual features.Source: §1 · p. 2
- The model predicts speech audio from silent video with a CNN, improving reconstructed-speech intelligibility and demonstrating reconstruction of out-of-vocabulary words.The authors also report that learning from the entire face rather than only the mouth greatly improves performance.Source: §1 · p. 2
2. SPEECH REPRESENTATION
The paper needs an acoustic representation that a neural network can estimate and that can be synthesized back into intelligible speech. It uses LPC and LSP features, with the LPC order kept low to isolate the effect of learned visual features.
2. SPEECH REPRESENTATION
- A suitable speech representation must be predictable by a neural network and synthesizable into intelligible audio; spectrogram magnitude lacks phase information, while raw waveform output lacked a suitable training loss.Source: §2 · p. 2
- LPC represents a speech signal’s spectral envelope, and LSP represents LPC coefficients in a form more stable and robust to quantization and small coefficient deviations.LSPs are described as useful for speech coding and transmission and well suited to this task.Source: §2 · p. 2
- The system downsamples audio to 8 kHz, uses overlapping 40 ms frames and 8th-order LPC followed by LSP decomposition, then concatenates two successive 9-element vectors into an 18-element sound vector.The authors chose the relatively low LPC order to isolate CNN-learned visual features from the hand-crafted features used by.Source: §2 · p. 2
3. PREDICTING SPEECH
The model predicts sound features for a video frame from a temporal clip of face frames, then synthesizes a waveform from those features. Using the full face rather than only the mouth yields substantially lower test error in the best configuration, while the waveform synthesis produces unvoiced speech.
3.1. Regressing sound features
- Each target sound-feature vector corresponds to one video frame, while the input is K consecutive grayscale face frames cropped and scaled to 128×128 pixels.The clip provides temporal context for interpreting lip movements that may be ambiguous in an isolated frame.Source: §3.1 · p. 2
- 40% lower test error for the full-face region than the mouth region at K = 9 shows the best reported configuration favors full-face input.The authors hypothesize that additional visual information helps disambiguate similar mouth movements.Source: §3.1 · pp. 2–3
- A VGG-like CNN with five convolution-and-pooling blocks and two fully connected layers predicts the 18-element sound representation using mean squared error loss.Source: §3.1 · p. 3
3.2. Generating a waveform
- Gaussian white noise substitutes for unpredicted excitation parameters during waveform synthesis, producing unvoiced, unnatural-sounding speech that the authors found adequate for intelligibility.The reconstructed spectral envelope is relatively accurate, but unvoiced excitation leaves out formants.Source: §3.2 · p. 3, Figure 3 · p. 3
4. EXPERIMENTS
The experiments test silent-video speech reconstruction on GRID using full-dataset and held-out-digit tasks, evaluated through human transcription. Reconstructed speech significantly outperformed, while held-out-digit results were promising relative to chance.
4.1. GRID corpus
- GRID contains 1,000 six-word sentences from 34 talkers, providing the corpus for the reconstruction experiments.The experiments used S4 videos for comparison with, while the results also included evaluation on S2.Source: §4.1 · pp. 3–4, Table 2 · p. 4
4.2. Sound prediction tasks
- For the full-dataset reconstruction task, the model used a random 80/20 train/test split of S4’s 1,000 videos, with all 51 GRID words represented in each set.The resulting representation vectors were converted back into waveforms using unvoiced excitation, and predicted-audio-only and original-video-plus-reconstructed-audio configurations were constructed.Source: §4.2 · p. 4
- The OOV task held out two distinct digits in each of five train/test splits and tested reconstruction on sequences containing those digits.For example, one split trained on digits 1–8 and tested on 9 and 0.Source: §4.2 · p. 4
4.3. Evaluating the speech predictions
- Twenty-three MTurk workers transcribed more than 400 three-second clips, classifying reconstructed words from the GRID vocabulary.The study included audio-only, audio-visual, and OOV audio-visual clips; listeners were unaware of clip type.Source: §4.3 · p. 4
4.4. Results
- Reconstructed speech was significantly more intelligible than in audio-only and audio-visual tests.Table 2 covers S2 and S4; the S2 listening test used combined audio and video, and S2's speech clarity was reported comparable to S4.Source: Table 2 · p. 4, §4.4 · p. 4
- Listeners were over five times more likely than chance to choose the correct held-out digit, supporting OOV reconstruction as a promising direction.The reported result averages digit annotation accuracies across five train/test splits; the authors suggest a larger, more diversified training vocabulary could increase OOV intelligibility.Source: §4.4 · p. 4, Table 3 · p. 4
5. CONCLUDING REMARKS
The work demonstrates the feasibility of reconstructing intelligible speech from silent video and reports promise for OOV word reconstruction. The authors identify less-constrained video and speaker generalization as directions for further research.
5. CONCLUDING REMARKS
- The work demonstrates the feasibility of reconstructing intelligible speech from silent video frames.It also reports OOV word reconstruction as promising when speechreading is modeled as regression and a CNN automatically learns relevant visual features.Source: §5 · p. 4
- OOV word reconstruction remains a promising direction using regression-based speechreading and CNN-learned visual features.Source: §5 · p. 4
- Further research includes testing real-world viability on less-constrained video and generalizing to speaker-independent and multiple-speaker reconstruction.Source: §5 · p. 4