Source-linked AI summary
Audio Word2Vec: Unsupervised Learning of Audio Segment Representations using Sequence-to-sequence Autoencoder
Yu-An Chung, Chao-Chung Wu, Chia-Hao Shen, Hung-Yi Lee, Lin-Shan Lee
TL;DR
The paper addresses how to represent variable-length audio segments as fixed-dimensional vectors that capture sequential phonetic structure without human annotation. It uses sequence-to-sequence autoencoders, including a denoising variant, and reports useful representations for query-by-example STD with better performance and lower computation than conventional DTW-based approaches.
Problem
Existing audio representations may not precisely capture sequential phonetic structure, while supervised approaches require labeled data and standard autoencoders require fixed-dimensional inputs.
Method
Sequence-to-sequence Autoencoder uses an encoder RNN to map an audio sequence to fixed-dimensional vector z and a decoder RNN to reconstruct the sequence; DSA adds denoising training.
Results
The learned vectors describe sequential phonetic structures and were useful for query-by-example STD, where the approach outperformed conventional DTW-based approaches at lower computation requirements.
Takeaways & Limitations
Audio Word2Vec provides an unsupervised fixed-dimensional representation for variable-length audio segments with demonstrated usefulness in query-by-example STD.
Abstract
from arXiv · showhide
The vector representations of fixed dimensionality for words (in text) offered by Word2Vec have been shown to be very useful in many application scenarios, in particular due to the semantic information they carry. This paper proposes a parallel version, the Audio Word2Vec. It offers the vector representations of fixed dimensionality for variable-length audio segments. These vector representations are shown to describe the sequential phonetic structures of the audio segments to a good degree, with very attractive real world applications such as query-by-example Spoken Term Detection (STD). In this STD application, the proposed approach significantly outperformed the conventional Dynamic Time Warping (DTW) based approaches at significantly lower computation requirements. We propose unsupervised learning of Audio Word2Vec from audio data without human annotation using Sequence-to-sequence Audoencoder (SA). SA consists of two RNNs equipped with Long Short-Term Memory (LSTM) units: the first RNN (encoder) maps the input audio sequence into a vector representation of fixed dimensionality, and the second RNN (decoder) maps the representation back to the input audio sequence. The two RNNs are jointly trained by minimizing the reconstruction error. Denoising Sequence-to-sequence Autoencoder (DSA) is furthered proposed offering more robust learning.
1. Introduction
The paper asks whether variable-length word audio can be mapped to fixed-dimensional vectors that preserve phonetic structure, without requiring labeled data. It proposes Audio Word2Vec using sequence-to-sequence autoencoding and evaluates it in query-by-example STD.
- Word2Vec motivates asking whether each word-level audio segment can likewise be transformed into a fixed-dimensional vector.
- Existing audio representations support speech applications but may not precisely describe sequential phonetic structures and were often designed heuristically.
- Supervised RNN representations require large amounts of labeled training data, limiting their use where annotations are scarce.
- Standard autoencoders learn representations unsupervised but traditionally require fixed-dimensional inputs, unlike arbitrary-length audio sequences.
- Audio Word2Vec uses Sequence-to-sequence Autoencoder representations learned from unannotated audio, with Denoising SA proposed for more robust learning.
- In preliminary query-by-example STD experiments, Audio Word2Vec was reported as more efficient than conventional DTW-based approaches while achieving better performance.
2. Proposed Approach
The proposed Sequence-to-sequence Autoencoder converts variable-length acoustic sequences into fixed-dimensional representations and reconstructs the inputs through a jointly trained recurrent decoder. A denoising variant trains on corrupted inputs while reconstructing the original sequences.
- 2. Proposed Approach: The approach maps an acoustic feature sequence of length T into a fixed-dimensional vector z ∈ R^d intended to describe sequential phonetic structure.
- 2. Proposed Approach: RNN hidden states provide temporal memory and support variable-length processing, while LSTM units address difficulty learning long-term dependencies.
- 2.2. Sequence-to-sequence Autoencoder (SA): An encoder RNN reads the input sequence and uses its final hidden state as the representation, while a decoder RNN generates an output sequence from that state.
- 2.2. Sequence-to-sequence Autoencoder (SA): SA trains the encoder and decoder jointly to reconstruct the input sequence, minimizing mean squared reconstruction error without labeled data.
- 2.3. Denoising SA: Denoising SA feeds a noise-corrupted sequence to the model and trains it to generate the original clean sequence.
3. An Example Application: Query-by-example STD
The paper applies learned audio vectors to unsupervised query-by-example spoken term detection, encoding archive segments and spoken queries into a shared fixed-dimensional space. Retrieval ranks archive segments by vector similarity with low online computation requirements.
- 3. An Example Application: Query-by-example STD: Archive audio is segmented at word boundaries and encoded into fixed-length vectors offline, while each incoming spoken query is encoded similarly.
- 3. An Example Application: Query-by-example STD: Query-by-example STD locates spoken query terms in a large spoken archive without speech recognition.
- 3. An Example Application: Query-by-example STD: The system ranks archive segments by cosine similarity between the query vector and the vectors of all archived segments.
- 3. An Example Application: Query-by-example STD: The online process has extremely low computation requirements because retrieval compares the query vector with precomputed archive vectors.
4. Experiments
Experiments evaluate whether unsupervised SA and DSA encode sequential phonetic structure and improve query-by-example STD retrieval. Across representation analyses and retrieval comparisons, DSA generally provides the strongest results.
- Experimental Setup: 5.4 hours of dev-clean data from 40 speakers trained the models, while 5.4 hours of test-clean data from 40 different speakers provided testing data.The experiments used 13-dimensional MFCCs and oracle word boundaries; query-by-example STD evaluated 5557 queries with MAP.
- Learned Representations: Larger phoneme sequence edit distances produced smaller average cosine similarities, while SA and DSA distinguished even one-phoneme differences.For zero edit distance, average similarity was 0.48–0.50, reflecting variation in acoustic realizations of identical phoneme sequences.
- Learned Representations: Words sharing suffixes remained clearly distinguishable in the learned vectors despite sequential encoding potentially emphasizing later acoustic features.For example, representations of “father” were closer to one another than to “mother,” while “another” was farther away.
- Learned Representations: Difference vectors for word pairs differing in their first or last phoneme were close in direction and magnitude, implying that phoneme replacement is represented in the vector space.The reported examples include new–few versus night–fight and days–day versus things–thing.
- Query-by-example STD: DSA achieved higher MAP than the naïve encoders after about 390 epochs, while SA surpassed them after about 450 epochs.DSA achieved much higher MAP than NE52, NE78, and NE104 and clearly outperformed SA in most cases; Figure 4 also includes DTW.
5. Conclusions and Future Work
The paper concludes that Sequence-to-sequence Autoencoder and its extension can learn unsupervised Audio Word2Vec representations that capture sequential phonetic structure and support query-by-example STD.
- SA and its extension learn Audio Word2Vec representations for audio segments without human annotation.
- The learned representations describe sequential phonetic structures and can support real-world applications such as query-by-example STD.
- Future Work: Future work includes training SA on larger corpora, exploring different dimensionalities and extensions, and evaluating other application scenarios.