PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Hidden Unit Contributions for Unsupervised Acoustic Model Adaptation

Pawel Swietojanski, Jinyu Li, Steve Renals

arXiv:1601.02828v2cs.CLcs.LGcs.SD

TL;DR

DNN acoustic-model adaptation must address speaker, channel, and environment mismatch between training and runtime data. The paper evaluates unsupervised LHUC and SAT-LHUC, which adapt hidden-unit contributions across four speech benchmarks, and reports consistent WER reductions across tasks and mismatch conditions.

  • Problem

    Acoustic-model adaptation addresses mismatch between training and runtime distributions caused by speaker variability, channel effects, and acoustic environments.

  • Method

    The paper evaluates unsupervised LHUC and SAT-LHUC, adapting speaker-dependent hidden-unit amplitudes while using small adaptation data and SAT training.

  • Results

    Across TED talks, AMI, Switchboard, and Aurora4, LHUC and SAT-LHUC produced 5–23% relative WER reductions depending on task and test mismatch.

  • Takeaways & Limitations

    LHUC works unsupervised with as little as 10s of data, complements SAT-fMLLR, and adapts sequence-trained DNNs using a cross-entropy adaptation objective.

Abstract

from arXiv · show

This work presents a broad study on the adaptation of neural network acoustic models by means of learning hidden unit contributions (LHUC) -- a method that linearly re-combines hidden units in a speaker- or environment-dependent manner using small amounts of unsupervised adaptation data. We also extend LHUC to a speaker adaptive training (SAT) framework that leads to a more adaptable DNN acoustic model, working both in a speaker-dependent and a speaker-independent manner, without the requirements to maintain auxiliary speaker-dependent feature extractors or to introduce significant speaker-dependent changes to the DNN structure. Through a series of experiments on four different speech recognition benchmarks (TED talks, Switchboard, AMI meetings, and Aurora4) comprising 270 test speakers, we show that LHUC in both its test-only and SAT variants results in consistent word error rate reductions ranging from 5% to 23% relative depending on the task and the degree of mismatch between training and test data. In addition, we have investigated the effect of the amount of adaptation data per speaker, the quality of unsupervised adaptation targets, the complementarity to other adaptation techniques, one-shot adaptation, and an extension to adapting DNNs trained in a sequence discriminative manner.

I. INTRODUCTION AND SUMMARY

DNN acoustic models improve speech recognition, but adapting them remains important because speakers, channels, and environments create training–runtime mismatch. The paper surveys feature-, model-, and hybrid-space adaptation approaches while motivating unsupervised DNN acoustic-model adaptation.

  • 10–32% WER reductions were reported for DNN acoustic models across varied tasks compared with discriminatively trained GMM systems.
  • Talker-adapted GMM systems with tandem or bottleneck features have often outperformed unadapted hybrid DNN/HMM systems, motivating DNN acoustic-model adaptation.
  • Acoustic-model adaptation addresses mismatch between training and runtime distributions caused by speaker variability, channel effects, and acoustic environments.
  • The paper investigates unsupervised LHUC and SAT-LHUC adaptation across TED talks, AMI, Switchboard, and Aurora4, including data quality, adaptation amount, complementarity, and sequence-trained models.
  • Feature-space methods such as fMLLR estimate speaker-specific affine input transforms, typically using a complete GMM system before DNN training and evaluation.
  • Model-space adaptation can modify DNN parameters directly, but adapting a large network creates very large speaker-dependent parameter sets and computational demands.

III. LEARNING HIDDEN UNIT CONTRIBUTIONS (LHUC)

LHUC treats hidden units as adaptive basis functions and adapts a trained network by changing their speaker-dependent amplitudes while keeping the basis functions fixed. The method uses gradient descent on unsupervised adaptation targets, with scaling-range control affecting transform capacity.

  • A neural network can approximate target functions by combining nonlinear adaptive basis functions weighted by learned coefficients.
  • LHUC keeps training-derived network basis functions fixed and adapts their combination for the target speaker or data distribution.
  • LHUC explicitly parameterizes each hidden unit’s amplitude with a speaker-dependent function in fully connected or post-pooling convolutional layers.
  • Figure 1 illustrates adapting a model optimized for f1 to f2 by recombining hidden-unit functions through LHUC scaling parameters.
  • LHUC adaptation learns speaker-specific amplitude parameters by gradient descent using targets supplied by adaptation data.
  • The scaling function’s range controls adaptation-transform capacity, which can help when unsupervised targets are noisy.

IV. SPEAKER ADAPTIVE TRAINING LHUC (SAT-LHUC)

SAT-LHUC addresses the limitation that test-only LHUC may start from poorly adaptable speaker-independent basis functions by jointly learning speaker-independent and speaker-dependent hidden-unit transforms. This produces a more tunable canonical model that can operate in speaker-dependent or speaker-independent modes.

  • Motivation: Test-only LHUC assumes speaker-independent basis functions provide a good starting point for adapting to the target distribution.The paper gives a counterexample where basis functions learned from competing training distributions poorly approximate adaptation data.
  • SAT-LHUC approach: SAT-LHUC jointly learns parallel speaker-dependent transforms and an additional average transform to model speaker-specific and shared properties.The approach estimates one transform per training distribution plus an extra transform for average properties.
  • SAT-LHUC approach: SAT-LHUC provides richer basis functions that fit data on average while capturing characteristics needed to reconstruct target training distributions.Figure 3 illustrates this using LHUC-0, LHUC-a, and LHUC-b transforms with shared basis functions.
  • Test-time adaptation: At test time, speaker-dependent LHUC parameters are inserted and optimized using unsupervised adaptation data, while adaptation methods require first-pass decoding targets.The shared model can also provide speaker-independent first-pass targets.
  • Model use: The jointly learned model is more tunable for unseen speakers and can be used in speaker-independent fashion to produce first-pass adaptation targets.Speaker-independent and speaker-dependent LHUC parameters are jointly optimized with the shared network parameters.
  • Training procedure: During SAT training, each example follows either a speaker-independent or speaker-dependent transform, with the split controlled by a Bernoulli probability γ.The SI/SD split may occur at speaker, utterance, or frame level, and mini-batch gradients sum contributions for each speaker.

V. EXPERIMENTAL SETUPS

The experiments evaluate LHUC and SAT-LHUC across four speech-recognition corpora covering talks, telephone conversations, meetings, and noise- and channel-robustness conditions. Configurations differ in acoustic conditions, feature types, adaptation targets, and whether adaptation clusters correspond to physical speakers.

  • Corpora: LHUC and SAT-LHUC are evaluated on TED talks, Switchboard, AMI meetings, and Aurora4.The study uses four corpora with distinct speech-recognition conditions.
  • Adaptation units: Adaptation may target a headset or conversation side rather than the physical speaker, so training clusters can differ from speaker counts.This distinction affects the number of transforms estimated during training.
  • TED: TED uses 143 hours of training speech from 813 talks and reports results with a 4-gram language model estimated from 751 million words.The experiments follow the IWSLT ASR evaluation protocol.
  • AMI: AMI compares Individual Headset Microphone and Single Distant Microphone conditions and also evaluates 40 mel-filter-bank features.The FBANK setup is relevant because fMLLR transforms are difficult to obtain for those features.
  • Switchboard: Switchboard evaluation uses eval2000, separating better-matched Switchboard speech from more mismatched CallHome English speech.The models use LDA/MLLT features and 3-gram language models estimated from Switchboard and Fisher data.
  • Aurora4: Aurora4 tests clean, noisy, channel-distorted, and noisy-channel-distorted speech across 14 conditions and SNRs from 5–15 dB.Training uses six noise types and multiple microphones under multi-condition training.

A. LHUC hyperparameters

The hyperparameter study examines which LHUC layers, adaptation iterations, and re-parameterization functions affect recognition performance. Results favor adapting more layers after starting with the bottom layer, while one adaptation sweep is generally sufficient and overfitting is limited.

  • Adaptation layers: Adapting only the bottom layer gives the largest initial WER drop, while adapting additional layers improves both LHUC and SAT-LHUC accuracy.Starting adaptation from the top layer is reported to be much less effective.
  • Adaptation iterations: One sweep over the adaptation data is sufficient, and additional iterations do not cause overfitting despite continued improvement of the adaptation objective.The finding suggests extensive model regularization is not necessary in this setting.
  • Re-parameterization: Test-only adaptation shows only a 0.1% absolute WER difference across re-parameterization functions despite larger frame-accuracy differences.This supports the reported robustness of LHUC against overfitting.
  • Re-parameterization: For SAT-LHUC training, a less constrained re-parameterization gives better WER for the speaker-independent model.Table II compares WER for different LHUC transform functions on TED tst2010 against a 15.0% unadapted baseline.
  • Optimization settings: The models use adaptation learning rate 0.8, while basic SI and SAT-LHUC training starts at 0.08 and later follows newbob learning-rate scheduling.These settings are shared across the reported model adaptations unless otherwise stated.

B. SAT-LHUC

SAT-LHUC trains hidden units to represent both average and distribution-specific characteristics, enabling more adaptable models for unseen speakers. Across TED experiments, segment- and frame-level training generally outperformed speaker-level training after adaptation.

  • SAT-LHUC trains hidden units to capture average and speaker-specific characteristics, making models more adaptable to unseen speakers.It estimates speaker-specific hidden unit amplitudes during training so the model can be tailored during later adaptation.
  • Speaker-level SAT-LHUC training produced the highest WERs for both SI and SD decodes.
  • Segment-level and frame-level SAT-LHUC training produced similar SI WERs, with frame-level training gaining 0.1% absolute after adaptation.
  • At γ = 0.5, adapted SAT-LHUC reduced WER by around 1% absolute, or 6% relative, versus the SI baseline test-only adapted LHUC model.Speaker-level training was around 0.4% absolute worse than segment- or frame-level training, largely because of poorer first-pass adaptation targets.
  • On TED, SAT-LHUC reduced WER by 0.3–1.1% absolute, or 2–6% relative, versus test-only LHUC; on AMI, it improved the DNN result by 4% relative.The gain was smaller for CNNs because max-pooling can select speaker-specific patterns at runtime.

C. Sequence model adaptation

The paper tests whether LHUC can adapt sequence-trained acoustic models without erasing their learned sequence information. On TED, LHUC delivered substantial unsupervised gains and compared favorably with a speaker-dependent-layer approach.

  • Adapting sequence-trained DNNs is challenging because a mismatched cross-entropy objective can erase sequence information through catastrophic forgetting.
  • LHUC adaptation of sMBR sequence-trained models reduced TED tst2013 WER by around 2% absolute, or 11% relative, for both SI and fMLLR SAT systems.The gain was similar to that obtained with cross-entropy models, and sequence representations were not apparently disrupted.
  • The IWSLT comparison used different training data and feature-preprocessing pipelines, so its results are not directly comparable with the TED results in Table VI.
  • Unsupervised LHUC improved TED tst2013 WER by 2.1% absolute, or 14% relative, versus 0.6% absolute, or 4% relative, for a speaker-dependent layer.The single-model LHUC system matched a more sophisticated postprocessing pipeline on that evaluation.
  • The authors conclude that LHUC can adapt sequence models unsupervised with a cross-entropy objective without removing learned sequence information.

D. Other aspects of adaptation

Additional experiments examine adaptation-data quantity, target quality, acoustic conditions, and one-shot reuse. LHUC benefits from very small unsupervised datasets, remains relatively insensitive to target quality, and improves both close- and distant-microphone conditions.

  • 10s of unsupervised adaptation data decreased WER by 0.5–0.8% absolute, while SAT-LHUC consistently outperformed LHUC across data amounts.SAT-LHUC’s advantage was roughly twofold up to 30s of adaptation data, then narrowed as duration increased.
  • Supervised adaptation targets further substantially decreased WER, with SAT-LHUC consistently outperforming LHUC.
  • LHUC was not very sensitive to adaptation-target quality across language-model rescoring conditions and adaptation-data durations.Only rescoring with the largest 4-gram LM produced much difference, and that improvement diminished in the final adapted system.
  • LHUC improved accuracy for both headset and distant-tabletop microphone data, although the gain was smaller for the distant-microphone condition.The distant-microphone system had twice as large WERs.
  • One-shot adaptation estimated LHUC transforms once for a held-out speaker and reused them across multiple meetings in a single-pass system.

E. Complementarity to feature normalisation

LHUC and SAT-LHUC complement feature-space normalization, with SAT-LHUC generally improving results further and recovering some degradation when adaptation data is scarce. Across TED, AMI, and Switchboard, the methods provide broad but task-dependent gains, with stronger relative performance under greater mismatch.

  • Complementarity with fMLLR: Feature-space fMLLR estimated from 10s of unsupervised data increased WER from 15.0% to 16.1% on TED tst2010.Combining fMLLR with LHUC or SAT-LHUC recovered part of this deterioration.
  • Complementarity with fMLLR: With 30s or more of adaptation data, fMLLR improved accuracy by around 1–2% absolute, while combining it with LHUC or SAT-LHUC added another 1% WER reduction.
  • Cross-task results: Across TED, AMI, and Switchboard, SAT-LHUC generally produced lower WER than LHUC and both methods were complementary with SAT-fMLLR training.
  • Cross-task results: On AMI, SAT-LHUC-trained DNNs matched the final results of the more complex SAT-fMLLR+LHUC system.
  • Cross-task results: On Switchboard, test-only LHUC reached 20.7% WER versus 21.7% for the SI system and 20.2% for SAT-fMLLR.Test-only LHUC remained below the SAT-fMLLR result, unlike the pattern observed on some other corpora.
  • Cross-task results: SAT-LHUC reached 20.3% WER, nearly matching SAT-fMLLR at 20.2%, while LHUC performed relatively better on the more mismatched Callhome subset.
  • Cross-speaker summary: LHUC or SAT-LHUC reduced WER for 89% of 200 TED, AMI, and Switchboard speakers, with SAT-LHUC consistently outperforming LHUC.

VII. LHUC FOR FACTORISATION

The factorised LHUC approach estimates separate speaker and environment transforms and interpolates them when joint adaptation data is unavailable. Joint transforms perform best on Aurora4, but factorisation remains more flexible, while visualisations show LHUC mainly changing non-speech representations across acoustic conditions.

  • Factorised adaptation: Speaker and environment transforms can be estimated separately and linearly interpolated with hyper-parameter α to form a combined transform.
  • Factorised adaptation: The method assumes the test environment is known so that the appropriate environmental transform can be selected.
  • Aurora4 results: On Aurora4, speaker-only and environment-only adaptation achieved 13.1% and 13.3% WER, respectively, versus 13.9% for the unadapted system.
  • Aurora4 results: Joint speaker-environment adaptation achieved 12.4% WER, while interpolated factorised adaptation with α = 0.7 reached 12.7% WER.Joint estimation was better, but factorised adaptation was more flexible when joint data was unavailable.
  • CNN experiments: The Maxout CNN experiments used multi-condition alignments as ground-truth labels, with a reported baseline of 10.9% versus 10.5% in prior work.
  • CNN experiments: tSNE visualisations showed phonetic-space normalisation across clean and noisy conditions, while non-speech frames retained condition-specific spaces and displayed most of LHUC’s effect.

VIII. CONCLUSIONS

The paper evaluates LHUC for unsupervised neural acoustic-model adaptation in test-only and speaker-adaptive-training settings across four speech corpora. It reports consistent WER improvements, complementarity with feature normalization, and extensions to small-data, sequence-trained, and factorised adaptation.

  • Conclusions: LHUC and SAT-LHUC produced 5–23% relative WER improvements across TED, AMI, Switchboard, and Aurora4, depending on task and mismatch.
  • Conclusions: LHUC worked unsupervised with as little as 10s of adaptation data and complemented feature-space normalization transforms such as SAT-fMLLR.
  • Conclusions: LHUC was also applied to sequence-trained DNN acoustic models using a cross-entropy adaptation objective.
  • Conclusions: Factorised adaptation estimated and interpolated separate transforms for the acoustic environment and speaker.
Loading 1601.02828v2…