Source-linked AI summary
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech
Yu-An Chung, James Glass
TL;DR
Existing speech embeddings largely emphasize acoustic-phonetic similarity, while semantic word embeddings are typically learned from transcribed text. Speech2Vec addresses this gap by combining an RNN Encoder-Decoder with skipgrams or cbow to learn fixed-length embeddings directly from word-level audio. Across the reported evaluation, its learned embeddings outperform Word2Vec embeddings learned from transcriptions.
Problem
Prior speech representations focused on acoustic-phonetic similarity, while learning semantic word embeddings generally required transcribing speech into text.
Method
Speech2Vec combines an RNN Encoder-Decoder with skipgrams or continuous bag-of-words to learn fixed-length embeddings directly from variable-length word audio.
Results
Speech2Vec embeddings outperform Word2Vec embeddings learned from transcriptions, with skipgrams Speech2Vec achieving the highest ρ in 8 of 13 benchmarks.
Takeaways & Limitations
Direct speech-based learning provides embeddings that capture semantic information while retaining information present in speech but absent from plain text.
Abstract
from arXiv · showhide
In this paper, we propose a novel deep neural network architecture, Speech2Vec, for learning fixed-length vector representations of audio segments excised from a speech corpus, where the vectors contain semantic information pertaining to the underlying spoken words, and are close to other vectors in the embedding space if their corresponding underlying spoken words are semantically similar. The proposed model can be viewed as a speech version of Word2Vec. Its design is based on a RNN Encoder-Decoder framework, and borrows the methodology of skipgrams or continuous bag-of-words for training. Learning word embeddings directly from speech enables Speech2Vec to make use of the semantic information carried by speech that does not exist in plain text. The learned word embeddings are evaluated and analyzed on 13 widely used word similarity benchmarks, and outperform word embeddings learned by Word2Vec from the transcriptions.
1. Introduction
Speech2Vec learns semantic word embeddings directly from raw speech, extending Word2Vec-style learning beyond transcribed text and its acoustic information loss.
- Motivation: Unlike prior speech-representation approaches focused on acoustic-phonetic similarity, Speech2Vec targets semantic similarity by modeling neighboring acoustic regions.Semantically similar underlying words are intended to occupy nearby locations in the embedding space.
- Contribution: Speech2Vec combines an RNN Encoder-Decoder with skipgrams or continuous bag-of-words to learn word embeddings from raw speech alone.The model handles audio segments corresponding to words without requiring other modalities.
- Evaluation: Speech2Vec embeddings are evaluated on widely used word-similarity benchmarks to assess whether they capture semantic information from spoken words.The stated objective is to learn fixed-dimensional representations whose geometry reflects word meaning.
- Motivation: Direct learning from speech avoids recognition errors introduced by transcription and can exploit prosody and other information absent from plain text.The authors motivate speech-based embeddings as a way to use richer signal information than transcripts provide.
- Contribution: The paper extends a preliminary Speech2Vec model with additional training methodologies, baseline comparisons, and systematic embedding analysis and visualizations.These additions broaden the evaluation and analysis of the proposed approach.
2. Proposed Approach
Speech2Vec encodes variable-length word audio into fixed-length vectors and trains those representations through neighboring speech segments using skipgrams or cbow.
- 2. Proposed Approach: Speech2Vec seeks fixed-length embeddings for variable-length acoustic sequences representing spoken words.The input may consist of features such as MFCCs, with sequence length T varying across audio segments.
- Architecture: Figure 1 pads all audio segments with zero vectors to a common length T and shares recurrent components across outputs or inputs depending on the training variant.Skipgrams shares the Decoder RNN across generated outputs, whereas cbow shares the Encoder RNN across encoded inputs.
- 2.2. Speech2Vec: The model uses an RNN Encoder-Decoder backbone and provides skipgrams and continuous bag-of-words training variants.Both variants are illustrated in Figure 1.
- Skipgrams: In skipgrams training, an input word segment is encoded into a fixed-dimensional vector that predicts nearby word segments within a context range k.The encoded vector is then used as the word embedding after training.
- Continuous bag-of-words: In cbow training, nearby segments are encoded by a shared Encoder, summed, and decoded to reconstruct the target segment.The resulting summed representation is taken as the target word embedding after training.
- 2.3. Differences between Speech2Vec and Word2Vec: Speech2Vec differs from Word2Vec by using Encoder and Decoder RNNs for variable-length acoustic sequences and by representing spoken instances of a word separately.Word2Vec instead uses a two-layer fully connected network and one deterministic vector per word.
3. Experiments
Experiments evaluated four Speech2Vec and Word2Vec variants on 13 word-similarity benchmarks, examining embedding size, corpus size, and representation variance. Skipgrams Speech2Vec generally performed best and became more stable for frequently occurring words.
- Evaluation setup: 13 word-similarity benchmarks measured Spearman’s rank correlation between model rankings and human rankings.The benchmarks cover different aspects of word similarity.
- Evaluation setup: Four models compared Speech2Vec skipgrams and cbow with Word2Vec skipgrams and cbow trained on LibriSpeech transcriptions.The Speech2Vec models used speech, while the Word2Vec counterparts used transcriptions.
- Embedding size: 50-dimensional embeddings captured sufficient semantic information for cbow Speech2Vec, skipgrams Speech2Vec, and cbow Word2Vec, whose best benchmark performance was mostly achieved at that size.Increasing embedding size did not always improve performance.
- Model comparison: Skipgrams Speech2Vec achieved the highest ρ in 8 of 13 benchmarks, outperforming cbow and skipgrams Word2Vec in combination.The authors suggest prosody may contribute semantic information absent from text.
- Model comparison: Skipgrams Speech2Vec consistently outperformed cbow Speech2Vec across all benchmarks and embedding sizes.The paper relates this result to the empirical behavior of skipgrams Word2Vec with small training corpora.
- Training corpus size: Performance was poor with 10% of the corpus and continued improving as training corpus size increased.Table 2 reports results using 50-dimensional embeddings; only six benchmarks were shown because of page limits.
- Variance study: For words occurring 5–99 times, cbow Speech2Vec vectors had lower variance, but skipgrams Speech2Vec vectors became lower-variance as occurrence counts increased.The gap favoring skipgrams grew with larger occurrence counts.
- Variance study: The deviation of skipgrams Speech2Vec decreased as word occurrence count increased, suggesting greater model stability.Variance was analyzed by averaging per-dimension standard deviations within occurrence-frequency groups.
4. Conclusions and Future Work
Speech2Vec extends Word2Vec to learn embeddings directly from speech, and its embeddings outperform transcription-based Word2Vec in the experiments. Future work includes speech-related extrinsic evaluation and less supervised word-boundary discovery.
- Speech2Vec integrates an RNN Encoder-Decoder with skipgrams or cbow to learn word embeddings directly from speech.
- The learned Speech2Vec embeddings outperform those produced by Word2Vec from transcriptions.
- Future evaluations will test the embeddings on machine listening comprehension and speech-based visual question answering.
- The current approach uses forced-alignment segmentations, motivating exploration of less supervised methods for learning word boundaries.