PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing wake words: Audio-visual Keyword Spotting

Liliane Momeni, Triantafyllos Afouras, Themos Stafylakis, Samuel Albanie, Andrew Zisserman

arXiv:2009.01225v1cs.CVeess.AS

TL;DR

The paper addresses detecting whether and when a user-specified keyword is spoken by a talking face, including when audio is absent or noisy. KWS-Net matches phonetic and visual sequences through a similarity map and detects alignment patterns in a zero-shot setting. It outperforms prior visual KWS on the reported benchmark, improves audio-visual KWS especially with noisy audio, and transfers to French and German with less language-specific data.

  • Problem

    The task is to detect and localize a user-specified word in talking-face video, including settings with unavailable or corrupted audio and visual ambiguities.

    Source: §1 · p. 2

  • Method

    KWS-Net encodes phonetic and visual sequences, forms a similarity map, and uses a CNN to detect alignment patterns for zero-shot keyword spotting.

    Source: §1 · p. 2, KWS-Net · p. 4

  • Results

    KWS-Net exceeds the previous visual KWS state of the art, improves audio-visual KWS for clean and noisy audio, and generalizes comparably to French and German with less language-specific data.

    Source: §1 · p. 2, §5.2 · p. 9, §6 · p. 10

  • Takeaways & Limitations

    Visual information can complement audio for keyword spotting, particularly when the audio signal is noisy, while the phoneme-based design supports transfer across languages.

    Source: §1 · p. 2, §5.2 · p. 9

    Visual KWS cannot distinguish words with different sounds but identical lip movements, such as ‘may,’ ‘pay,’ and ‘bay,’ without audio.

    Source: §1 · p. 2

Abstract

from arXiv · show

The goal of this work is to automatically determine whether and when a word of interest is spoken by a talking face, with or without the audio. We propose a zero-shot method suitable for in the wild videos. Our key contributions are: (1) a novel convolutional architecture, KWS-Net, that uses a similarity map intermediate representation to separate the task into (i) sequence matching, and (ii) pattern detection, to decide whether the word is there and when; (2) we demonstrate that if audio is available, visual keyword spotting improves the performance both for a clean and noisy audio signal. Finally, (3) we show that our method generalises to other languages, specifically French and German, and achieves a comparable performance to English with less language specific data, by fine-tuning the network pre-trained on English. The method exceeds the performance of the previous state-of-the-art visual keyword spotting architecture when trained and tested on the same benchmark, and also that of a state-of-the-art lip reading method.

1 Introduction

Keyword spotting detects a user-specified word in continuous speech, using audio, visual, or both streams, rather than recognizing unrestricted phrases. KWS-Net targets zero-shot visual spotting, motivated by applications with absent or noisy audio and constrained by visual ambiguities.

1 Introduction

  • KWS detects a user-specified word in continuous speech, unlike ASR or AVSR, which recognize phrases or sentences without a specified target.The specified keyword can make KWS simpler and potentially higher-performing than unrestricted recognition.Source: §1 · p. 1
  • Visual KWS supports silent-film browsing, noisy-audio wake-word recognition, human-robot interaction, and assistance for people with speech impairment or aphonia [34].Source: §1 · p. 2
  • Visual KWS must handle words with identical visemes, such as ‘may,’ ‘pay,’ and ‘bay,’ as well as accents, speech-rate variation, mumbling, imaging changes, and co-articulation.Source: §1 · p. 2
  • KWS-Net uses a similarity map to match phoneme and viseme sequences, then detects alignment patterns to determine whether and when a keyword occurs.Source: §1 · p. 2
  • KWS-Net exceeds prior visual KWS performance, improves audio-visual spotting for clean and noisy audio, and generalizes to French and German with less language-specific data.Source: §1 · p. 2

2 Related Work

Related work spans visual speech recognition and audio keyword spotting. KWS-Net is situated between these areas by using visual sequence representations while targeting a user-specified keyword.

Lip reading

  • Visual speech recognition methods include CTC models, seq2seq models with attention, transformers, and hybrid CTC-seq2seq approaches.Source: §2 · pp. 2–3

Audio KWS

  • Audio KWS research includes HMMs, fully connected networks, time-delay networks, CNNs, graph CNNs, RNNs, convolutional-recurrent models, and seq2seq methods.Source: §2 · p. 3

Visual KWS

  • Visual KWS-Net computes a similarity map from viseme and phonetic sequence embeddings, then uses a CNN classifier to detect the alignment pattern and produce keyword probabilities over time.Source: Figure 2 · p. 3

KWS-Net

KWS-Net encodes a talking-face clip and a queried keyword, matches their temporal representations through a similarity map, and classifies alignment patterns. The architecture supports zero-shot keyword detection and localization, with optional boundary supervision and audio-visual variants.

KWS-Net

  • The visual model encodes video frames and the keyword’s phoneme sequence, computes their dot-product similarity map, and applies a CNN to detect keyword-alignment patterns.The map has dimensions np × tv, while the CNN collapses the phoneme dimension and subsamples time.Source: KWS-Net · p. 4
  • The CNN outputs a keyword probability at each temporal location; the maximum probability determines sequence-level presence and its frame position estimates the keyword location.Source: KWS-Net · p. 4
  • Concatenating the phonetic embedding with the similarity map helps represent short phonemes and keywords containing repeated phonemes that may violate a simple diagonal alignment.Source: KWS-Net · p. 4
  • Training uses binary cross-entropy on clip-keyword pairs, optionally restricting the positive-sample maximum to known keyword boundaries for localization supervision.Source: KWS-Net · pp. 4–5

Differences to prior work

  • KWS-Net treats spotting as object detection over similarity-map alignments and trains the CNN architecture jointly end to end instead of using probabilistic word-level HMMs.Source: KWS-Net · p. 5
  • Direct phoneme encoding with a variable-length BiLSTM better reflects pronunciation, supports multiple pronunciations, and enables sharing the encoder across languages.Source: KWS-Net · p. 5

4 Experiments

Experiments evaluate KWS-Net across audio-visual datasets, unseen-keyword settings, baselines, ablations, and training procedures, using ranking and detection metrics. The setup also tests fine-grained localisation and language generalisation on French and German data.

Datasets

  • Experiments use LRW, LRS2, LRS3, and French and German LRS3 subsets, with dataset statistics covering splits, utterances, word instances, duration, vocabulary, and examples.LRS3 is more challenging than LRS2 because it includes wider viewpoints, microphones or headsets, and more frequent head movements.Source: §4 · pp. 5–6, Table 1 · p. 6

Test setup

  • Training and evaluation use keywords with n_p >=6 phonemes, while standard experiments keep test keywords disjoint from the training vocabulary to evaluate unseen words.Language-generalisation experiments relax the disjoint-vocabulary constraint, allowing test keywords to have appeared during training.Source: §4 · p. 6

Baselines

  • The study compares KWS-Net with a prior KWS model, a P2G variant, a visual-ASR system, and a query-by-example retrieval method with different comparability and vocabulary constraints.The visual-ASR baseline is described as state of the art on LRS2 using publicly available training data, while the query-by-example method supports only a closed word set.Source: §4 · p. 6

Ablations

  • Ablations remove word time boundaries, the shortcut phonetic embedding, or the BiLSTM keyword encoder in favour of a P2G encoder-decoder.These variants test localisation supervision, the phonetic shortcut, and the keyword representation.Source: §4 · p. 7

Pre-training and fine-tuning

  • The visual feature extractor is initialized from a word-level lip-reading model, frozen during training, and followed by curriculum training on LRW before fine-tuning on sequence-level datasets.Features are pre-computed to accelerate training.Source: §4 · p. 7

Evaluation

  • Evaluation searches every test clip with held-out query words, uses unbalanced positive and negative clips, applies stride-one classification for fine-grained localisation, and reports R@N, mAP, and EER.Metrics are averaged with standard deviations over the last five checkpoints after convergence; noisy-audio experiments add babble noise at 0 dB SNR during training half the time.Source: §4 · p. 7

5 Results

Across visual-only, audio-visual, and multilingual settings, KWS-Net improves keyword spotting through similarity-map alignment and remains effective with noisy audio and limited language-specific data.

5.1 Visual-only KWS-Net

  • 37.9% R@1, 53.9% mAP, and 5.7% EER improve over the P2G baseline’s 30.0% R@1, 43.5% mAP, and 6.3% EER on LRS2.KWS-Net also exceeds the lip-reading baseline in R@5 and mAP, although visual-ASR has higher R@1.Source: §5.1 · p. 7, Table 2 · p. 8
  • Longer keywords are easier to spot, phrase queries raise R@1 from 37.9% to 65.3%, and fine-tuning on seen words only marginally improves zero-shot performance.The reported seen-versus-unseen EER changes from 5.7% to 5.1%.Source: §5.1 · p. 9, Table 3 · p. 9

5.2 Audio-visual KWS-Net

  • 72.2% R@1 with clean audio and 52.7% at 0 dB SNR show that visual information improves clean and noisy audio KWS over 67.7% and 27.6% audio-only R@1.Across -10 dB to 20 dB SNR, the audio-visual model surpasses both video-only and audio-only KWS-Net.Source: §5.2 · p. 9, Table 4 · p. 9

5.3 Extension to other languages: French and German

  • 48.7% mAP on profile views versus 60.6% on frontal clips shows robustness to side views but better performance on frontal LRS3 clips.LRS3 is described as more challenging than LRS2 for visual KWS.Source: §5.3 · p. 10
  • Audio-visual KWS-Net outperforms audio-only and visual-only models across English, French, and German, despite less language-specific data for some languages.Visual-only EER is 13.0% for German versus 8.4% for French, while the French training set is five times larger.Source: §5.3 · p. 10, Table 5 · p. 10

6 Conclusion

KWS-Net is a CNN-based visual keyword-spotting architecture inspired by object detection methods. It exceeds prior visual KWS performance, benefits from audio in clean and noisy conditions, and generalises beyond English.

  • KWS-Net exceeds the previous state of the art on the LRS2 dataset in visual-only keyword spotting.Source: §6 · p. 10
  • Combining audio and visual modalities improves keyword spotting for both clean and noisy audio.Source: §6 · p. 10
  • KWS-Net generalises to languages other than English.Source: §6 · p. 10

A.1 KWS-Net Architecture

The appendix specifies architecture details for audio-only and visual-only KWS-Net, including convolutional notation and input dimensions.

  • Table A.1 details the audio-only and visual-only KWS-Net architectures.Source: Table A.1 · p. 14
  • K and S denote convolutional kernel widths and strides, while mp and avgp denote max- and average-pooling layers.Source: Table A.1 · p. 14
  • The notation also identifies keyword phoneme count, video-frame count, and residual convolution blocks.Source: Table A.1 · p. 14

A.2 Baseline Architecture

The baseline encodes the keyword’s phoneme sequence, combines it with visual features, and predicts keyword presence per frame with an auxiliary decoder.

  • The baseline architecture encodes the query keyword phoneme sequence into a compact embedding and concatenates it with the visual feature sequence.Source: §A.2 · p. 15, Figure A.2 · p. 15
  • A BiLSTM and fully connected layers produce a binary prediction for each frame.Source: §A.2 · p. 15
  • An auxiliary decoder predicts the keyword’s grapheme representation from the phoneme embedding and is trained with an auxiliary loss for regularisation.Source: §A.2 · p. 15
  • The experiments reverse the original phoneme-grapheme input-output order, using P2G instead of G2P, which improved performance in the authors’ experiments.Source: §A.2 · p. 15

A.3 Datasets

The datasets differ in whether test keywords are unseen during training, with strict vocabulary separation for LRW and LRS2 but not for the LRS3 variants.

  • LRW and LRS2 use disjoint train-validation and test vocabularies for zero-shot evaluation with unseen keywords.Source: §A.3 · p. 15
  • LRS3, LRS3-Fr, and LRS3-De2 do not impose disjoint vocabularies between training and testing.Source: §A.3 · p. 15
  • Table A.2 reports the full keyword vocabulary sizes used for training and testing in each dataset.Source: §A.3 · p. 15

A.4 Training curriculum

The training curriculum specifies dataset vocabulary handling and optimization settings, while additional results examine audio-visual fusion and qualitative keyword detection examples.

A.4 Training curriculum

  • The model is trained for 100 epochs with Adam, using learning-rate schedules of 10^-3 for LRW pre-training and 10^-4 for LRS2 fine-tuning.Batches contain positive clip-keyword pairs and an equal number of randomly selected negative keywords.Source: §A.4 · p. 15

A.5 Additional results

  • Table A.3 evaluates visual-only, audio-only, and audio-visual KWS-Net on LRS2 under clean audio and 0 dB noisy audio.Source: Table A.3 · p. 16
  • Audio-visual fusion performs best with audio weight w_audio = 0.7 and video weight w_video = 0.3, surpassing an audio-only model on clean LRS3 audio.Source: §A.5 · p. 16, Table A.4 · p. 16

A.6 Qualitative examples

  • The appendix provides qualitative examples and time-varying detection probabilities for visual-only, audio-only, and audio-visual KWS-Net on positive LRS2 clips.Source: Figure A.3 · p. 17, §A.6 · p. 17

Related papers

Loading 2009.01225v1…