Source-linked AI summary
Towards Learning a Universal Non-Semantic Representation of Speech
Joel Shor, Aren Jansen, Ronnie Maor, Oran Lang, Omry Tuval, Felix de Chaumont Quitry, Marco Tagliasacchi, Ira Shavitt, Dotan Emanuel, Yinnon Haviv
TL;DR
Speech lacks a standard benchmark for comparing general-purpose representations on non-semantic tasks, limiting systematic evaluation of transfer learning. The paper introduces NOSS and TRILL, a self-supervised representation learned from AudioSet using temporal proximity, and evaluates them across diverse downstream and personalization tasks. TRILL outperforms other compared representations and, after fine-tuning, reaches state-of-the-art or competitive performance on many tasks.
Problem
Speech lacks a standard benchmark for comparing general-purpose representations across diverse non-semantic tasks.
Method
The paper introduces NOSS and TRILL, a self-supervised speech representation learned from AudioSet using temporal proximity as its training signal.
Results
TRILL performs best among compared representations, while fine-tuning achieves or exceeds previously reported performance on many NOSS tasks.
Takeaways & Limitations
NOSS and TRILL provide a publicly released benchmark and representation for evaluating and transferring non-semantic speech models across low-resource, personalization, and medical tasks.
Takeaways & Limitations
The reported study uses a small subset of VoxCeleb1 filtered according to YouTube’s privacy guidelines.
Abstract
from arXiv · showhide
The ultimate goal of transfer learning is to reduce labeled data requirements by exploiting a pre-existing embedding model trained for different datasets or tasks. The visual and language communities have established benchmarks to compare embeddings, but the speech community has yet to do so. This paper proposes a benchmark for comparing speech representations on non-semantic tasks, and proposes a representation based on an unsupervised triplet-loss objective. The proposed representation outperforms other representations on the benchmark, and even exceeds state-of-the-art performance on a number of transfer learning tasks. The embedding is trained on a publicly available dataset, and it is tested on a variety of low-resource downstream tasks, including personalization tasks and medical domain. The benchmark, models, and evaluation code are publicly released.
1. Introduction
The paper introduces NOSS, a diverse benchmark for evaluating general-purpose non-semantic speech representations, and TRILL, a self-supervised representation designed for transfer across such tasks. TRILL performs strongly across the benchmark, while fine-tuning achieves state-of-the-art results on many tasks and supports compact on-device models.
- Benchmark: NOSS evaluates speech representations on diverse non-semantic tasks, including emotion, speaker, language, medical, and personalization settings.The benchmark uses publicly available tasks and adds intra-speaker sub-tasks to measure personalization.
- Representation: TRILL learns speech embeddings self-supervised from AudioSet using temporal proximity as a proxy objective.Segments closer in time are encouraged to be closer in embedding space.
- Results: Fine-tuning TRILL sets a new state-of-the-art on many previously published tasks.The paper reports that fine-tuning further boosts results.
- Results: A single unsupervised representation performs best on the benchmark when compared with existing feature-based and learned representations.This comparison evaluates general usefulness across the benchmark rather than performance on only one task.
- Deployment: The original and distilled TRILL models are released, with the distilled version intended for on-device inference and training.The paper presents model release as part of its contributions.
2. Background
The background motivates transfer learning through reusable representations and standardized evaluation across heterogeneous downstream tasks. It situates speech representation learning within a broad literature spanning supervised, semi-supervised, unsupervised, and AudioSet-based approaches.
- Transfer learning: Transfer learning representations improve sample efficiency by extracting information from raw signals before task-specific learning.This strategy has been applied across multiple domains.
- Evaluation: Standard benchmarks should contain varied downstream tasks so representations can be evaluated across different properties of a domain.Vision and NLP already provide examples of such benchmarks.
- Adaptation: Representations can be adapted through frozen linear classifiers, full fine-tuning, or multi-task learning.Fine-tuning generally matches or outperforms fully supervised downstream models, especially with limited labeled data.
- Related work: Audio representation learning includes supervised, semi-supervised, unsupervised, metric-learning, predictive-coding, and autoencoding methods.The related literature includes methods such as L3, AuDeep, APC, CPC, and metric learning.
- Data: AudioSet provides a large and variable general-purpose audio dataset that can support learning embeddings for multiple speech tasks.The paper characterizes AudioSet as an audio analogue of ImageNet.
3. Non-Semantic Speech Benchmark (NOSS)
NOSS standardizes evaluation of non-semantic speech representations across diverse inter-speaker tasks and personalization-oriented intra-speaker tasks.
- Inter-speaker tasks: NOSS evaluates representations on diverse non-semantic tasks, including paralinguistic recognition, speaker identification, language identification, and medical diagnosis.The datasets differ in target classes and example counts.
- Intra-speaker tasks: The benchmark includes intra-speaker tasks to test personalization and adaptation using data from individual speakers.Accuracy is averaged over speakers, with speakers used for both training and testing.
- Intra-speaker tasks: Intra-speaker evaluation is meaningful for CREMA-D, SAVEE, and Speech Commands, but not for speaker-dependent labels such as language, medical diagnosis, or speaker identity.The latter tasks lack meaningful intra-speaker versions because training and testing on the same speaker would not assess task adaptation.
4. Experiments
The experiments learn TRILL from temporal proximity in speech and compare it with classical and learned representations on NOSS using shallow downstream models, fine-tuning, and distillation.
- TRILL representation: TRILL uses temporal proximity as self-supervision because non-semantic speech attributes change more slowly than phonetic and lexical content.The representation is trained with a triplet-loss objective using L2 distance, hinge loss, and a nonnegative margin.
- TRILL representation: TRILL is trained on speech-tagged AudioSet clips using 0.96-second log-mel spectrogram windows and a 10-second temporal parameter.The training setup primarily distinguishes segments from the same clip versus different clips.
- Other representations: The comparison includes Mel spectrograms, OpenSmile, YAMNet, VGGish, and a randomly initialized network.These baselines cover classical feature-based and learned representations.
- Evaluation: Small downstream models use logistic regression, random forests, or LDA, with task-specific aggregation, normalization, and representation choices.Performance is evaluated on canonical or repeated random splits, including speaker-averaged intra-speaker evaluation.
- Fine-tuning and distillation: Fine-tuning addresses limited adaptation capacity in frozen embeddings, while early stopping is used for relatively small benchmark datasets.The experiments also include intra-speaker fine-tuning and distillation into a smaller MobileNet model.
5. Results
TRILL performs strongly across the benchmark, with fine-tuning improving most tasks and distillation largely preserving the larger model’s performance, including on personalized tasks.
- Benchmark results: TRILL outperforms previous results on three of six benchmark tasks and is competitive with the previous best on two of the remaining three.On those two tasks, the previous best results use additional visual or textual modalities.
- Benchmark results: Among compared representations, TRILL performs best on five of six benchmark tasks and two of three intra-speaker tasks.The comparison is summarized across tasks using a linear regression of observed accuracies with model and task as explanatory variables.
- Distillation: The distilled model matches the larger model on personalized tasks and shows no performance degradation on five of nine tasks.It has statistically insignificant degradation on one task and minor degradation on the remaining tasks.
6. Analysis
Fine-tuning TRILL improves performance across most tasks, while effective representations are often found in intermediate rather than final layers. Per-speaker fine-tuning generally improves personalization, though its effect varies across speakers.
- Fine-tuning the final embedding gives a clear boost on most tasks and reaches state-of-the-art performance on 3 of 6 datasets.The paper presents fine-tuning as effective for adapting a strong learned representation to downstream speech tasks.
- Effective representations from both YAMNet and TRILL occur in intermediate layers rather than at their final layers.The final layers may discard information useful for non-semantic tasks, including temporally varying properties.
- Intermediate layers of all learned representations substantially outperform their respective top layers on Speech Commands.
- Per-speaker fine-tuning generally improves accuracy, increasing performance for 31 speakers, leaving 49 mostly unchanged, and decreasing it for 12.
7. Conclusions
The paper introduces NOSS and TRILL to evaluate and learn broadly useful non-semantic speech representations. Fine-tuning TRILL performs strongly across tasks, while the released models and code support further use, including on-device deployment.
- NOSS provides a benchmark for fairly comparing speech representations on diverse non-semantic and personalization tasks.
- TRILL uses a self-supervised training criterion and performs well across all benchmark tasks.
- Fine-tuning TRILL on a small amount of data outperforms or matches almost all previously reported NOSS results and surpasses other representations.
- The benchmark, evaluation code, TRILL, and TRILL-distilled are publicly available, with little or no performance loss after distillation.