Source-linked AI summary
Deep Multimodal Learning for Audio-Visual Speech Recognition
Youssef Mroueh, Etienne Marcheret, Vaibhava Goel
TL;DR
The paper addresses how to fuse audio and visual speech information for AV-ASR, particularly whether visual input helps when audio is clean. It evaluates separate-modality feature fusion and bilinear networks that model cross-modal correlations, with posterior combination producing further PER reduction. The reported conclusion is that visual input improves phonetic classification in clean conditions, while bilinear modeling yields additional error-rate reduction.
Problem
AV-ASR requires effective integration of audio and visual information, and the paper examines whether visual input helps recognition even under clean speech conditions.
Method
The paper trains separate audio and visual DNNs for feature fusion, then introduces bilinear softmax networks for joint training and modality-correlation modeling.
Results
34.03% final PER is obtained by combining bilinear-network posteriors with those from the fused model, versus 35.83% PER for the fused model.
Takeaways & Limitations
Visual input improves phonetic classification even with clean audio, and bilinear modeling leads to further phone error-rate reduction.
Takeaways & Limitations
The bilinear model uses a regularization constraint ||Uj||F ≤ λ for numerical stability, with λ set as a regularization parameter.
Abstract
from arXiv · showhide
In this paper, we present methods in deep multimodal learning for fusing speech and visual modalities for Audio-Visual Automatic Speech Recognition (AV-ASR). First, we study an approach where uni-modal deep networks are trained separately and their final hidden layers fused to obtain a joint feature space in which another deep network is built. While the audio network alone achieves a phone error rate (PER) of $41\%$ under clean condition on the IBM large vocabulary audio-visual studio dataset, this fusion model achieves a PER of $35.83\%$ demonstrating the tremendous value of the visual channel in phone classification even in audio with high signal to noise ratio. Second, we present a new deep network architecture that uses a bilinear softmax layer to account for class specific correlations between modalities. We show that combining the posteriors from the bilinear networks with those from the fused model mentioned above results in a further significant phone error rate reduction, yielding a final PER of $34.03\%$.
1 Introduction
The paper investigates deep multimodal fusion for AV-ASR, emphasizing that visual information can improve recognition even with clean audio. It evaluates separate-modality fusion and bilinear models for capturing cross-modal correlations.
- Visual speech information supports recognition in noise and can disambiguate voices with similar acoustics, while the paper focuses on its value under clean conditions.
- AV-ASR combines audio recordings and talking-person videos, but integrating both modalities into effective recognition models is challenging.
- The study trains separate audio and visual networks, concatenates their final hidden-layer outputs, and trains another classifier on the resulting joint representation.
- The paper also introduces bilinear networks that model modality correlations through joint training and combines their posteriors with the fused model.
- The methods are evaluated on the IBM AV-ASR Large Vocabulary Studio Dataset, with the paper organized around feature extraction, fusion, bilinear modeling, and posterior combination.
2 Audio-Visual Data Set & Feature Extraction
The IBM AV-ASR Studio Dataset provides clean audio-visual recordings from hundreds of speakers, with synchronized audio and video features extracted for phonetic classification.
- The dataset contains 40 hours of clean studio recordings from 262 speakers, using 16 KHz audio and 30-frame-per-second video.
- The recordings use 704 × 480 video resolution and a 10,400-word vocabulary, with a two-hour test set from 22 speakers.
- Audio features comprise 24 MFCC coefficients, stacked over nine frames, projected to 40 dimensions with LDA, and expanded with ±4 context frames into 360-dimensional vectors.
- Visual processing detects faces and mouths, extracts scattering coefficients from 64 × 64 mouth regions, and replicates frames to match audio timing.
- Each audio-video frame receives one of 1328 context-dependent phoneme targets, and classification error over these classes is reported as PER.
3 Uni-modal DNNs & Feature Fusion
The first fusion approach trains audio and visual DNNs separately, concatenates their final hidden representations, and trains a classifier in the fixed fused space.
- Separate audio and visual DNNs are trained with cross-entropy, then their final hidden-layer outputs are concatenated into a joint audio-visual representation.
- The fused feature space has dimension 400, using individual networks with low-dimensional final hidden layers to control representation size.
- 41.25% PER is achieved by the audio network alone, compared with 69.36% for the visual network alone.
- 35.77% PER results from a deep classifier in the fused space, while a softmax-only classifier achieves 35.83% PER.
- The fused audio-visual representation substantially improves PER under clean audio conditions, with deep and shallow fusion performing roughly on par.
4 Bilinear Deep Neural Network
The bilinear bimodal DNN jointly models audio and visual modalities through class-specific correlations in a factored fused representation. Weight sharing across phoneme groups reduces the parameters required for the joint representation.
- Bilinear fusion: The bilinear bimodal DNN fuses audio and visual networks at the decision function through a bilinear term.The intermediate layers remain separate, while the final hidden-layer representations are combined to model cross-modal correlations.
- Factored bilinear softmax: Factorizing the bilinear term reduces computational complexity as the number of classes increases.The factorization uses modality-specific projections into a lower-dimensional fused space motivated by Canonical Correlation Analysis.
- Factored bilinear softmax: For fixed modality projections, the model learns a linear classifier in a fused space formed by element-wise products of projected audio and visual features.The projections define CCA-like lower-dimensional spaces, and their element-wise product supplies the softmax input.
- Factored bilinear softmax with sharing: Sharing bilinear weights across leaves with the same parent models correlations at the phoneme level rather than for every contextual state.In AV-ASR, 1,328 contextual phoneme states are grouped under 42 phoneme categories.
- Factored bilinear softmax with sharing: Weight sharing reduces the joint-representation parameters from C × F to G × F.The shared weights are tied among labels belonging to the same group.
5 Back-propagation with the Factored Bilinear DNN with Sharing
The factored bilinear DNN with sharing is trained by propagating label-level and group-level errors through the two modality networks. Updates include standard back-propagation and Frobenius-norm projection for stability.
- Error propagation: Training examples carry both leaf-state labels and parent phoneme-group labels, so the shared model tracks errors at both levels.The label error is represented over C states, while the group error is represented over G phoneme groups.
- Parameter updates: The bilinear softmax parameters are updated using gradients associated with the class or shared group representations.The formulation defines the parameter matrix W from group-specific vectors and computes gradients for the bilinear softmax.
- Error propagation: Errors propagated to the audio and visual networks include messages passed through the bilinear interaction.After these cross-modal terms, the remaining updates follow standard back-propagation in both networks.
- Parameter updates: Each parameter follows an update θ ← θ + η ∂E/∂θ, where η is the learning rate.The same update form is applied across the variables in the back-propagation procedure.
- Parameter updates: The projection matrices U1 and U2 are followed by projection onto a Frobenius ball to control their norms.This constraint is applied after the gradient step for numerical stability.
- Parameter updates: Without sharing, the update rules are similar, with δG replaced by δL.
6 Combining Posteriors from Bimodal and Bilinear Bimodal Networks
The experiments evaluate several factored bilinear architectures and posterior combinations on the IBM AV-ASR Studio dataset. Combining bimodal and bilinear bimodal posteriors produces the strongest reported gain.
- Architecture evaluation: 35.83% PER is achieved by the separately trained bimodal DNN baseline.Each individual bilinear architecture does not improve on this baseline.
- Architecture evaluation: Averaging posteriors from the three bilinear architectures yields a small gain over the individual architectures.
- Posterior combination: 1.8% absolute gain results from averaging posteriors of the bimodal and bilinear bimodal networks.The paper attributes this improvement to uncorrelated errors between the bilinear and bimodal networks.
7 Conclusion
The paper studies deep multimodal phonetic classification from audio and visual inputs and finds that visual information improves performance even under clean acoustic conditions. Its bilinear bimodal DNN further reduces the error rate by leveraging modality correlations.
- Visual information significantly improves phonetic classification when added to speech in clean acoustic conditions.
- The bilinear bimodal DNN leverages correlations between audio and visual modalities for further error-rate reduction.