PaperlayerView exact revision on arXiv

Source-linked AI summary

BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation

Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, Kunio Kashino

arXiv:2103.06695v2eess.AScs.LGcs.SD

TL;DR

Many audio self-supervised methods rely on relationships among segments from different times, which can be problematic for repeated or brief sounds. This paper introduces BYOL-A, which learns from two augmented views of a single audio segment and achieves state-of-the-art results across downstream tasks using normalization and audio-specific augmentations.

  • Problem

    Relationships among multiple audio segments can be contradictory, motivating representation learning without relying on temporal agreement or disagreement.

  • Method

    BYOL-A extends BYOL to audio by learning from augmented view pairs derived from one segment, using normalization, mixup, and random resize crop.

  • Results

    BYOL-A with 2,048 dimensions outperforms other methods in all tasks, averaging 77.8% versus 68.7% for COLA with 2,048 dimensions.

  • Takeaways & Limitations

    Single-segment representation learning is possible and can outperform methods based on agreement or disagreement among multiple audio segments.

Abstract

from arXiv · show

Inspired by the recent progress in self-supervised learning for computer vision that generates supervision using data augmentations, we explore a new general-purpose audio representation learning approach. We propose learning general-purpose audio representation from a single audio segment without expecting relationships between different time segments of audio samples. To implement this principle, we introduce Bootstrap Your Own Latent (BYOL) for Audio (BYOL-A, pronounced "viola"), an audio self-supervised learning method based on BYOL for learning general-purpose audio representation. Unlike most previous audio self-supervised learning methods that rely on agreement of vicinity audio segments or disagreement of remote ones, BYOL-A creates contrasts in an augmented audio segment pair derived from a single audio segment. With a combination of normalization and augmentation techniques, BYOL-A achieves state-of-the-art results in various downstream tasks. Extensive ablation studies also clarified the contribution of each component and their combinations.

I. INTRODUCTION

Previous audio methods often learn from relationships among segments at different times, but such relationships can be contradictory. BYOL-A instead learns general-purpose representations from one segment by creating augmented views focused on foreground sounds and texture details.

  • Motivation: Different-time segment relationships can be misleading because repetitive sounds may recur remotely while short events may occur unpredictably.These problems also affect contrastive and triplet-loss methods because their objectives compare multiple samples.
  • Method: The augmentation design targets foreground acoustic events and sound-texture details as complementary parts of a general-purpose representation.Foreground consistency is encouraged through random background variation, while pitch shifting and time stretching preserve useful details across variations.
  • Contribution: BYOL-A learns general-purpose audio representations from a single audio segment without expecting relationships between different time segments.This avoids the multiple-segment comparisons used by many previous audio methods.
  • Method: Normalization, mixup, and random resize crop form the core augmentation strategy, with pre- and post-normalization supporting stable augmentation and statistical correction.The paper also reports extensive ablation studies examining the contribution of each augmentation block.
  • Method: BYOL-A applies BYOL to audio by creating two augmented views from one segment and minimizing loss between their projected embeddings.The online network is updated by the loss, while the target network follows the online network through an exponential moving average.

II. RELATED WORK

Related audio methods commonly exploit temporal relationships or contrastive comparisons, while BYOL avoids negative samples and supports learning from a single input. BYOL-A also differs from prior mixup-based approaches in using mixup specifically to model audio content.

  • BYOL: BYOL avoids negative samples by minimizing the mean squared error between embeddings of augmented views from the same input.Its architecture and training algorithm are designed to avoid collapsed representations while claiming state-of-the-art performance.
  • Mixup methods: Prior mixup-based methods such as DACL, i-MIX, and mix-back remain based on contrastive loss comparing positive and negative samples.The paper identifies this contrastive objective as a fundamental difference from BYOL-A.
  • Audio representation learning: CPC, Jansen et al., TRILL, and COLA learn representations using relationships among segments cropped from different times.These methods use contrastive or triplet objectives to structure representations according to temporal proximity or clip membership.
  • BYOL-A: The paper distinguishes BYOL-A from prior approaches by combining single-input learning with audio augmentations designed around foreground and content information.The supplied figure captions identify the BYOL-A system and its audio augmentation and random-resize-crop modules.

B. Bootstrap Your Own Latent (BYOL)

BYOL learns representations by matching augmented views of the same input without negative samples. An online network is trained against a slowly moving target network, with a predictor helping avoid collapse.

  • BYOL creates two augmented views of one input and matches their online and target representations without using negative samples.The online network produces a representation, projection, and prediction, while the target network processes the second view.
  • The online network uses weights θ, whereas the target network has the same architecture with weights ξ.
  • BYOL minimizes mean squared error between L2-normalized online predictions and target projections.
  • The loss is symmetrized by reversing which augmented view enters the online and target networks.
  • Only θ is optimized directly; ξ is updated as an exponential moving average, ξ ← τξ + (1 − τ)θ.
  • The online predictor and moving-average target update encourage information encoding and avoid constant collapsed representations.

III. BYOL FOR AUDIO (BYOL-A)

BYOL-A adapts BYOL to learn general-purpose audio representations from a single segment rather than relationships among multiple time segments. Its audio augmentations target foreground events and content details, with normalization supporting stable processing.

  • III. BYOL FOR AUDIO (BYOL-A): BYOL-A learns general-purpose audio representations from a single audio segment without expecting relationships between different time segments.
  • III. BYOL FOR AUDIO (BYOL-A): BYOL-A replaces BYOL’s image augmentation module with an audio module that creates contrasting augmented views from log-scaled mel-spectrograms.
  • III. BYOL FOR AUDIO (BYOL-A): The augmentation module contains four blocks, beginning with pre-normalization and duplicating the normalized input before mixup.
  • III. BYOL FOR AUDIO (BYOL-A): Mixup creates contrast for foreground acoustic-event representations, while Random Resize Crop approximates pitch shifting and time stretching for content details.
  • III. BYOL FOR AUDIO (BYOL-A): Input normalization uses the training-sample mean µ and standard deviation σ, producing a standardized representation for subsequent processing.
  • III. BYOL FOR AUDIO (BYOL-A): Normalization stabilizes augmentation by reducing parameter sensitivity and statistical differences between training datasets.

B. Mixup for foreground acoustic event

BYOL-A’s mixup augmentation varies background content while preserving the main sound, encouraging foreground-event invariance. Random Resize Crop supplies complementary pitch and time variations for learning content details.

  • B. Mixup for foreground acoustic event: Mixup adds a small amount of a randomly selected past input, making the added audio part of the mixed sample’s background.
  • B. Mixup for foreground acoustic event: Contrasting background sounds between paired mixup outputs encourages representations invariant to background variation while retaining foreground acoustic events.
  • B. Mixup for foreground acoustic event: BYOL-A applies mixup only to audio features, converting log-scaled inputs to linear scale before mixing and back afterward.
  • B. Mixup for foreground acoustic event: The mixing ratio λ is sampled from U(0.0, α), and smaller α values preserve more of the original input contents.
  • B. Mixup for foreground acoustic event: The mixing counterpart x_k is randomly selected from a FIFO memory bank containing 2,048 past inputs.
  • C. RRC for all content details: Random Resize Crop approximates pitch shifting and time stretching so details are learned despite differences in pitch and time across outputs.
  • C. RRC for all content details: RRC samples crop regions from a virtual boundary that can extend beyond the input, then resizes them to the input dimensions.

D. Post-Normalization for statistical drift adjustment

Post-normalization adjusts statistical drift introduced by earlier augmentations so BYOL-A’s final views approximate a standard normal distribution. The resulting representations are evaluated across six downstream audio tasks using frozen pretrained encoders.

  • D. Post-Normalization for statistical drift adjustment: Post-normalization adjusts augmentation-induced statistical drift so final BYOL-A views become approximately N(0, 1).It uses means and standard deviations calculated from batch samples.
  • IV. EXPERIMENTS: BYOL-A representations are evaluated on six audio downstream tasks with a linear model trained on frozen BYOL-A encoder embeddings.
  • IV. EXPERIMENTS: BYOL-A is pretrained on AudioSet, a large-scale dataset commonly used in previous studies.
  • IV. EXPERIMENTS: Pretraining and evaluation are repeated three times for full AudioSet and five times for 1/10 AudioSet or FSD50K, with results averaged.
  • IV. EXPERIMENTS: Audio is converted to 16,000-Hz log-mel spectrograms with 64-ms windows, 10-ms hops, and 64 mel bins.
  • IV. EXPERIMENTS: Pretraining uses randomly cropped 96-frame segments corresponding to 1,014 ms, while downstream segment lengths follow each dataset’s average duration.

2) Encoder network:

The encoder uses a compact CNN with a tunable representation dimension, and the study evaluates it across six downstream audio classification tasks.

  • Encoder architecture: BYOL-A uses a simple CNN based on a DCASE 2020 audio-captioning solution.The authors describe this as a realistic design choice because smaller networks are used in audio machine learning.
  • Encoder architecture: The representation embedding dimension is varied as a hyper-parameter and also sets the linear-layer sizes.This variation tests how representation size affects performance.
  • Encoder architecture: The projection and prediction heads use MLPs with a 4,096-unit linear layer and 256-dimensional output embeddings.They include batch normalization and ReLU, with Adam optimization at learning rate 0.0003.
  • Pretraining setup: Pretraining uses 1,963,807 unlabeled AudioSet samples, while ablations use a 1/10 subset containing 210,315 samples.The full AudioSet and subset settings support the main experiments and ablation studies, respectively.
  • Evaluation tasks: The downstream evaluation covers musical instrument, acoustic scene, speaker, language, and command classification tasks.The tested datasets are NSynth, UrbanSound8K, VoxCeleb1, VoxForge, Speech Commands V2, and SPCV2/12.
  • Evaluation tasks: SPCV2/12 uses ten words plus silence and others, creating class imbalance and greater complexity than SPCV2.The setup has fewer classes but includes aggregated and background-noise-derived categories.

5) Linear evaluation details:

The study evaluates frozen BYOL-A encoders with linear classifiers and compares them against prior representation-learning methods under matched experimental settings.

  • Linear evaluation: Linear evaluation trains one supervised linear layer on frozen encoder embeddings extracted from downstream audio samples.Each evaluation runs ten times and reports averaged accuracy, with early stopping during 200-epoch maximum training.
  • Comparison setup: COLA’ reproduces COLA with an added normalization module and the same encoder as BYOL-A, enabling comparison of single- and two-segment inputs.Its pretraining settings match BYOL-A except for a batch size of 1,024.
  • Comparison setup: Table I compares TRILL, COLA, OpenL3, COALA, COLA’, and BYOL-A across the downstream tasks.BYOL-A and COLA’ are also tested with 512-, 1,024-, and 2,048-dimensional embeddings.
  • Results: BYOL-A with 2,048 dimensions outperforms the other methods in all tasks.The reported average is 77.8%, compared with 68.7% for COLA’ with 2,048 dimensions.
  • Results: BYOL-A with 512-dimensional embeddings also shows competitive performance, especially on the speech command tasks.This indicates that the strongest reported configuration is not the only competitive embedding size.

C. Ablation study: Contribution of data augmentations

Ablations show that mixup and Random Resize Crop contribute differently, with their combination producing the strongest average performance among tested augmentation settings.

  • 1) Contribution of mixup compared with Gaussian-noise: Mixup improves the average result 3.9 points, from RRC’s 68.4% to Mixup+RRC’s 72.3%.The combined configuration is evaluated with 512-dimensional BYOL-A pretrained on 1/10 AudioSet.
  • 1) Contribution of mixup compared with Gaussian-noise: Adding Gaussian noise to Mixup+RRC degrades average performance 1.8 points, down to 70.5%.The authors interpret this as evidence that within-dataset mixup is more effective in this setting than interpolation with random data points.
  • 2) Contribution of mixup, RRC, and their combination: Mixup alone reaches 63.9% average accuracy, whereas Gaussian noise alone reaches 23.2%.Mixup produces especially large gains on the SPCV2 speech-command task.
  • 2) Contribution of mixup, RRC, and their combination: RRC alone reaches 68.4%, while Mixup+RRC reaches 72.3%, the best average among the tested combinations.The combination is complementary on average, except for a performance drop on VC1 speaker identification.

D. Ablation study: Contribution of normalization blocks

Normalization supports BYOL-A by stabilizing mixup-related computations and correcting augmentation-induced statistical drift, and both normalization blocks contribute to performance.

  • Normalization ablation: Removing pre-normalization degrades performance by −1.8 to −3.4, exceeding the −0.2 degradation from removing post-normalization.The ablation removes either block but retains the other to preserve the basic setup.
  • Normalization ablation: Pre-normalization stabilizes log-mixup-exp by making the log-mel spectrogram range constant.Without it, the mixup α sweet spot shifts, and α values down to 0.05 do not recover 72.3% performance.
  • Normalization ablation: Normalization and augmentation together contribute to performance gain and recovery.The conclusion attributes BYOL-A’s overall result to the modules working together as one augmentation module.

APPENDIX

The appendix details the encoder and evaluates BYOL-A augmentation blocks, comparisons with COLA’+, and pretraining on FSD50K alongside AudioSet.

  • Encoder architecture: The appendix describes the encoder architecture and its fixed-dimensional representation outputs across 512-, 1,024-, and 2,048-dimensional settings.The corresponding parameter counts are 600,192, 1,649,792, and 5,321,856, respectively.
  • Augmentation blocks: Mixup improved COLA’+ accuracy by 5.8, compared with 1.8 for Gaussian noise.COLA’+ extends COLA’ with augmentation blocks, whereas the original COLA does not use data augmentations.
  • Augmentation blocks: Adding RRC to COLA’+Mixup improved performance by 1.5, versus an 8.4 improvement when RRC was applied to BYOL-A with Mixup only.The appendix attributes the smaller gain to resizing and cropping segments that were already randomly cropped.
  • Augmentation blocks: The BYOL-A augmentation module was effective with COLA’+, but more effective with BYOL-A’s single-segment input setting.This comparison is summarized across the augmentation-block experiments.
  • Pretraining datasets: Additional experiments compare BYOL-A pretraining on FSD50K with pretraining on AudioSet.The FSD50K development subset contains 40,966 samples, compared with 210,315 samples in the AudioSet 1/10 subset, and uses 500 pretraining epochs.
Loading 2103.06695v2…