Source-linked AI summary
Deep convolutional acoustic word embeddings using word-pair side information
Herman Kamper, Weiran Wang, Karen Livescu
TL;DR
The paper addresses how to represent variable-length whole-word speech as fixed-dimensional embeddings that discriminate word types. It compares CNN approaches and introduces a Siamese CNN trained from known word pairs with a hinge loss. The Siamese and classifier CNNs substantially improve over earlier word-discrimination results, while the Siamese model requires weaker supervision.
Problem
Whole-word speech systems need fixed-dimensional acoustic embeddings that accurately distinguish word types without relying solely on phonetic-unit modelling.
Method
The paper compares CNN-based embeddings and trains a Siamese CNN with tied networks, word-pair side information, and a hinge-like loss separating same-word from different-word pairs.
Results
0.549 AP: the best Siamese CNN improves over 0.365 AP for prior whole-word embeddings and 0.469 AP for DTW with learned frame features; the word classifier reaches 0.532 AP.
Takeaways & Limitations
Siamese CNNs provide strong acoustic word embeddings with weaker pairwise supervision, while retaining performance comparable to a more strongly labelled word classifier.
Takeaways & Limitations
The comparison uses a slightly different training set from the prior work, although the authors report comparable training-set size.
Abstract
from arXiv · showhide
Recent studies have been revisiting whole words as the basic modelling unit in speech recognition and query applications, instead of phonetic units. Such whole-word segmental systems rely on a function that maps a variable-length speech segment to a vector in a fixed-dimensional space; the resulting acoustic word embeddings need to allow for accurate discrimination between different word types, directly in the embedding space. We compare several old and new approaches in a word discrimination task. Our best approach uses side information in the form of known word pairs to train a Siamese convolutional neural network (CNN): a pair of tied networks that take two speech segments as input and produce their embeddings, trained with a hinge loss that separates same-word pairs and different-word pairs by some margin. A word classifier CNN performs similarly, but requires much stronger supervision. Both types of CNNs yield large improvements over the best previously published results on the word discrimination task.
1. INTRODUCTION
The paper revisits whole words as speech-modelling units and evaluates fixed-dimensional acoustic word embeddings for word discrimination. Its Siamese CNN uses word-pair side information and achieves strong performance despite weaker supervision.
- Motivation: Whole-word models map variable-length speech segments into fixed-dimensional embeddings that can be compared directly.This supports speech recognition, indexing, and query-by-example applications.
- Evaluation: The paper compares CNN-based approaches with the best earlier acoustic embedding method on a word discrimination task.The task evaluates embedding accuracy without training a complete recognition or search system.
- Contribution: The proposed Siamese CNN learns from known same-word and different-word pairs using a hinge-like contrastive loss.Its tied networks produce embeddings for paired speech segments and separate pairs by a margin.
- Results: The Siamese CNN outperforms the best earlier approach and performs similarly to a word-classifier CNN despite weaker supervision.A compact embedding obtained through post-processing preserves best performance.
2. ACOUSTIC WORD EMBEDDING APPROACHES
Acoustic word embeddings map whole-word speech into fixed-dimensional vectors, with methods ranging from reference-vector and supervised classifier approaches to weakly supervised Siamese CNNs. The proposed Siamese design uses tied CNNs and pairwise similarity objectives, while bottlenecks and dimensionality reduction target compact representations.
- Embedding objective: An acoustic word embedding function maps variable-length speech into a fixed-dimensional vector space where same-word segments are close and different-word segments are separated.Distances between embeddings provide the basis for word-type discrimination.
- Reference vector methods: Reference-vector methods represent a target segment through DTW costs to reference exemplars before dimensionality reduction.Their main drawback is the need to compute many DTW alignments.
- Word classification CNN: A word-classification CNN predicts word types from padded speech and uses its softmax representation as an embedding.The output dimensionality equals the vocabulary size, and convolutional and pooling layers reduce padding effects.
- Word similarity Siamese CNNs: Siamese CNNs use same-type word pairs as weaker supervision, with tied networks producing embeddings directly from padded speech inputs.Pairs not in the same-type set are assumed to belong to different types.
- Word similarity Siamese CNNs: The cosine hinge loss requires same-type pairs to be more similar than different-type pairs by a margin.This relative-distance objective matches discrimination decisions based on comparisons among pairs.
- Design goals: Siamese networks are introduced as a new approach for obtaining acoustic word embeddings from speech, while bottleneck layers support adjustable compact representations.The paper seeks embeddings that are both discriminative and low-dimensional.
3. EXPERIMENTS
The experiments evaluate CNN-based acoustic word embeddings on same-different word discrimination, comparing them with prior DTW and embedding approaches. Siamese CNNs, especially with a hinge loss, achieve strong performance with weaker paired supervision and remain effective at compact dimensionalities.
- Evaluation setup: The evaluation uses same-different word discrimination, computing cosine distances between embeddings of word pairs and summarizing threshold performance with average precision.Models are trained on Ytrain and evaluated on a disjoint Ytest set from Switchboard speech.
- Baselines: Models 1–3 use DTW over frame-level representations, while model 4 is the best previous whole-word acoustic embedding approach.Model 3 is the best previously reported result overall among the compared prior systems, and model 4 is the strongest prior whole-word embedding system.
- Neural models: Convolutional processing substantially improves AP over the corresponding non-convolutional neural model under the same word-label supervision.The comparison between models 5 and 6 isolates the benefit of convolutional layers, while both use Wtrain supervision comparable to model 4.
- Neural models: A lower-dimensional bottleneck version of the convolutional model still improves substantially over the prior whole-word embedding baseline.Model 7 reduces embedding dimensionality while retaining a sizable margin over model 4.
- Siamese CNNs: The lcos hinge Siamese CNN outperforms its lcos cos2 counterpart and substantially improves over the previous best whole-word embedding approach.Its performance is similar to the word-classification CNN despite using weaker pairwise supervision rather than fully labelled word classes.
- Dimensionality and supervision: The lcos hinge Siamese CNN remains strongest across tested development-set dimensionalities, while LDA produces smaller embeddings without performance loss when word labels are available.Compact embeddings are relevant for downstream tasks requiring smaller representations; the hinge loss is aligned with relative same-versus-different discrimination.
4. CONCLUSION
The paper studies CNN-based fixed-dimensional acoustic word embeddings for whole-word speech segments and compares them on word discrimination. A Siamese CNN trained with hinge loss gives the strongest new result, while a word-classifier CNN performs similarly but requires stronger supervision.
- Contribution: CNNs map whole-word speech segments to fixed-dimensional acoustic word embeddings for discrimination between word types.The study compares several CNN-based embedding approaches on the same-different word discrimination task.
- Results: 0.549 AP is achieved by the hinge-loss Siamese CNN, exceeding 0.365 AP for prior whole-word embeddings and 0.469 AP for DTW with learned frame features.These are test-set average precision results on the same-different word discrimination task.
- Results: The word-classifier CNN reaches 0.532 AP but requires much stronger labelled supervision and performs worse at smaller dimensionalities.The Siamese approach instead uses known word pairs as side information.
- Future work: Future work targets sequence models and downstream applications including term discovery, speech recognition, and search.The paper identifies these as subsequent application areas for the learned embeddings.