Source-linked AI summary
Adversarial synthesis based data-augmentation for code-switched spoken language identification
Parth Shastri, Chirag Patil, Poorval Wanere, Shrinivas Mahajan, Abhishek Bhatt, Hardik Sailor
TL;DR
Code-switched speech data is difficult to obtain, limiting data available for spoken language identification. This work uses GANs to generate Hindi-English Mel spectrograms for augmentation, reporting gains in UAR and F1 score over baseline augmentation methods.
Problem
Code-switched speech data is hard to find, and the work addresses under-representation of the Hindi-English code-switched class in spoken LID.
Source: §I · p. 1Method
The work trains a conditional GAN on code-switched Mel spectrograms, using log F0 contours, to generate samples for classifier augmentation.
Source: §I · p. 1, B. Method Description · p. 3, §VI · p. 7Results
4.7% increase in UAR and 8.1% increase in F1 score were observed compared with baseline methods used to augment raw audio and spectrograms.
Source: §VI · p. 8Takeaways & Limitations
The generated code-switched samples were used to validate GAN-based augmentation with a three-class spoken-LID classifier.
Source: §VI · p. 8
Abstract
from arXiv · showhide
Spoken Language Identification (LID) is an important sub-task of Automatic Speech Recognition(ASR) that is used to classify the language(s) in an audio segment. Automatic LID plays an useful role in multilingual countries. In various countries, identifying a language becomes hard, due to the multilingual scenario where two or more than two languages are mixed together during conversation. Such phenomenon of speech is called as code-mixing or code-switching. This nature is followed not only in India but also in many Asian countries. Such code-mixed data is hard to find, which further reduces the capabilities of the spoken LID. Hence, this work primarily addresses this problem using data augmentation as a solution on the on the data scarcity of the code-switched class. This study focuses on Indic language code-mixed with English. Spoken LID is performed on Hindi, code-mixed with English. This research proposes Generative Adversarial Network (GAN) based data augmentation technique performed using Mel spectrograms for audio data. GANs have already been proven to be accurate in representing the real data distribution in the image domain. Proposed research exploits these capabilities of GANs in speech domains such as speech classification, automatic speech recognition, etc. GANs are trained to generate Mel spectrograms of the minority code-mixed class which are then used to augment data for the classifier. Utilizing GANs give an overall improvement on Unweighted Average Recall by an amount of 3.5% as compared to a Convolutional Recurrent Neural Network (CRNN) classifier used as the baseline reference.
I. INTRODUCTION
The introduction addresses the scarcity of code-switched speech data by proposing GAN-based synthesis for augmentation, using Mel spectrograms as the representation. It also outlines GAN training and the WGAN-GP variant used to address training issues.
I. INTRODUCTION
- The work proposes GANs to generate under-represented code-switched samples and uses the generated spectrograms as additional training data for a CRNN classifier.It frames this use of GANs as a novel approach to augmenting code-switched samples.Source: §I · pp. 1–2
- Code-switched data is scarce, and synthesis-based augmentation is proposed as a way to expand speech data while more faithfully reflecting speech-rate variation than simple speed changes.The study focuses on Hindi-English speech and treats Mel spectrograms as image-like inputs for GAN training.Source: §I · p. 1
II. GANS: AN OVERVIEW
- GAN training is described as a min-max game in which the generator learns the data distribution through the discriminator’s ability to distinguish real from synthetic inputs.The generator and discriminator are trained simultaneously.Source: §II · p. 2
- WGAN minimizes Earth-Mover distance to mitigate GAN training issues, while WGAN-GP adds a gradient penalty to enforce the discriminator’s 1-Lipschitz constraint.The paper describes its method as conditional and says it follows the original training algorithm with the same hyperparameters.Source: §II · p. 2
A. Motivation
The study targets limited Indic code-switched data and evaluates GAN-based augmentation. It trains a conditional GAN on code-switched Mel spectrograms, using log F0 conditioning and Wasserstein loss with gradient penalty and reconstruction loss.
A. Motivation
- Only the under-represented code-switched class is augmented, with a GAN used to increase effective training data through representation learning.The proposed spoken LID system takes 2-D Mel spectrograms as input; its CNN captures spatial invariance and its Bi-LSTM captures sequential temporal context.Source: A. Motivation · p. 2, B. Method Description · p. 3
B. Method Description
- The conditional DCGAN uses a log F0 contour as conditioning information for generating Mel spectrograms.The log F0 contour is extracted using the WORLD algorithm; the architecture adapts SpecGAN for Mel-spectrogram generation.Source: B. Method Description · p. 3
- The generator projects the conditioning vector and uses five upsampling convolutional layers, while the critic combines the conditioning vector and spectrogram before classification.The conditioning vector is reshaped and concatenated with the spectrogram along the channel axis before the critic's convolutional layers.Source: B. Method Description · p. 3
- Training optimizes Wasserstein loss with gradient penalty and adds reconstruction loss to guide the networks.The reconstruction-loss weight is λrecon = 10.Source: B. Method Description · p. 4
C. Details of implementation
The implementation converts and scales Mel spectrograms and log F0 inputs, then trains the network with specified normalization, activation, optimizer, and iteration settings.
D. Details of implementation
- Spectrograms are min-max scaled to [-1, 1] and sized 128 × 128; log F0 is converted to semitones, without a stated normalization range.The FFT window is 1024 with a 16 ms overlap; the log F0 contour has length 128.Source: C. Details of implementation · p. 4
- The generator uses batch normalization and tanh output, while the critic uses layer normalization and LeakyReLU with α = 0.2.Generator dropout is p = 0.5; Adam uses learning rate 1×10^-4, β1 = 0.5, and β2 = 0.9.Source: C. Details of implementation · p. 4
- Training runs for 150k iterations with batch size 8 and converges around 115k iterations.Source: C. Details of implementation · p. 4
D. Architecture Details
The proposed architecture modifies SpecGAN with conditional log F0 input and distinct generator and critic design choices.
D. Architecture Details
- The architecture conditions generation on a log F0 contour, uses layer normalization in the critic and batch normalization with upsampling convolutions in the generator, and omits phase shuffle.Noise is implicitly supplied through dropout in the generator's initial layers rather than directly feeding a latent vector.Source: D. Architecture Details · p. 4
E. Dataset for training the GAN
The GAN is trained on Hindi-English code-mixed recordings from the MUCS ASR Challenge dataset, while test data is reserved for classification. Its FID evaluation compares generated and real samples, and the reported FID decreases as training iterations increase.
E. Dataset for training the GAN
- The cleaned and silence-trimmed training set contains approximately 23 hours of Hindi-English code-mixed speech, and the test set contains approximately 4 hours.The recordings are 16 kHz, 16-bit PCM WAV files; training data is used for GAN generation and test data for classification.Source: E. Dataset for training the GAN · p. 4
F. Evaluation of GAN
- FID uses CRNN-baseline features to compare real and generated samples as multivariate Gaussian distributions.The penultimate CRNN convolution layer supplies features; lower FID indicates better generated-sample quality.Source: F. Evaluation of GAN · p. 4
- FID decreases as GAN training iterations increase for n = 2400.The paper also compares FID for different sample sizes of real and generated images against the real test data.Source: Figure 3 · p. 5, F. Evaluation of GAN · p. 5, Table III · p. 5
IV. EXPERIMENTS
The experiments assemble a three-class Hindi, English, and Hindi-English dataset and represent audio as normalized Mel spectrograms for CRNN classification.
A. Language identification dataset
- The small dataset combines English, Hindi, and Hindi-English examples borrowed from existing Indian English, Hindi, and code-mixed speech resources.The code-mixed examples come from MUCS ASR Challenge test data; the combined dataset is resampled to 16 kHz and contains about six hours of audio.Source: A. Language identification dataset · p. 5
- Figure 4 juxtaposes real test-data Mel spectrograms and their normalized log-F0 contours with spectrograms generated by the proposed GAN.Source: Figure 4 · p. 5
B. Classification and feature extraction
- The CRNN performs three-class spoken LID on Hindi, English, and Hindi-English using normalized 128×128 Mel spectrograms from audio segments of about two seconds.The spectrograms use 128 Mel filterbanks, a 16 ms overlap window, and values normalized to [0, 1].Source: B. Classification and feature extraction · pp. 5–6
C. Train-Test data Split
The experiments reserve separate test data, simulate scarcity in the Hindi-English class, and compare the proposed approach with four reference baselines.
C. Train-Test data Split
- The Hindi-English speech comes from technical spoken tutorials, with sentence timestamps used to extract utterances and a separate subset reserved for testing.Training and validation use an 80%-20% split, with five-fold cross-validation or evaluation for model selection.Source: C. Train-Test data Split · p. 6
D. Dataset imbalance
- The imbalance heuristic randomly removes 80% of Hindi-English samples to approximate real-world under-representation.Table IV is identified as showing the number of examples per class after this heuristic.Source: D. Dataset imbalance · p. 6, Table IV · p. 6
E. Refernce baselines
- Four reference baselines comprise no augmentation, SpecAugment, raw-audio time-stretching, and raw-audio pitch-shifting.SpecAugment applies time and frequency masking; time-stretch rates range from 0.5 to 1.5, and pitch shifts range from −4 to 4 semitones.Source: E. Refernce baselines · p. 6
F. Evaluation
Evaluation uses accuracy, precision, F1-score, and Unweighted Average Recall; the paper notes that F1-score and UAR are more reliable than accuracy for imbalanced classification.
F. Evaluation
- The system is evaluated with accuracy, precision, F1-score, and Unweighted Average Recall (UAR).Source: F. Evaluation · p. 6
- F1-score is the harmonic mean of precision and recall, and the paper identifies F1-score and UAR as more reliable than accuracy for imbalanced classification.Source: F. Evaluation · p. 6
- The UAR equation combines true-positive and false-positive rates weighted by the positive and negative class proportions.The text defines the positive and negative totals as p and n, respectively.Source: F. Evaluation · p. 6
G. Training Method
The proposed method improves UAR over the baseline and is trained with Adam and early stopping. The reported comparison also includes F1-score and class-wise precision and recall.
G. Training Method
- The classifier uses Adam with a learning rate of 1 × 10^-5 and trains for about 30 epochs with early stopping.Evaluation is on a balanced test dataset.Source: G. Training Method · p. 6
V. PERFORMANCE AND RESULTS
- 4.7% UAR improvement over the baseline is reported for the proposed method, alongside an 8.1% relative F1-score improvement over the baselines.Time-stretch improves UAR by 0.8%; the proposed model also performs better than the other models without hurting the other two classes' scores.Source: §V · p. 7
VI. CONCLUSION
The proposed GAN augments minority Hindi-English code-switched Mel spectrograms using the audio sample’s log F0 contour. The authors report improved classifier performance over baseline augmentation methods and describe further directions for GAN-based speech generation.
VI. CONCLUSION
- The GAN augments minority Hindi-English code-switched audio Mel spectrograms using the sample’s log F0 contour.The generated samples were evaluated with FID, which decreased as GAN training progressed.Source: §VI · pp. 7–8
- 4.7% higher UAR and 8.1% higher F1 scores were observed compared with baseline methods augmenting raw audio and spectrograms.The comparison concerns the augmented minority Hindi-English class.Source: Figure 6 · p. 7, Table VI · p. 8, §VI · p. 8
- The authors identify higher-resolution spectrograms, progressive growing, Style-GANs, CycleGANs, and phonetic conditioning as directions for improving GAN effectiveness.They also propose applying the technique to ASR using phonetic information to produce realistic speech utterances.Source: §VI · p. 8