PaperlayerView exact revision on arXiv

Source-linked AI summary

BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation

Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, Kunio Kashino

arXiv:2103.06695v2eess.AScs.LGcs.SD

TL;DR

Many audio self-supervised methods rely on relationships among segments from different times, which can be problematic for repeated or brief sounds. This paper introduces BYOL-A, which learns from two augmented views of a single audio segment and achieves state-of-the-art results across downstream tasks using normalization and audio-specific augmentations.

  • Problem

    Relationships among multiple audio segments can be contradictory, motivating representation learning without relying on temporal agreement or disagreement.

    Source: §I · p. 1

  • Method

    BYOL-A extends BYOL to audio by learning from augmented view pairs derived from one segment, using normalization, mixup, and random resize crop.

    Source: §I · p. 2, §V · p. 7

  • Results

    BYOL-A with 2,048 dimensions outperforms other methods in all tasks, averaging 77.8% versus 68.7% for COLA with 2,048 dimensions.

    Source: B. Experimental results: Comparison… · p. 5

  • Takeaways & Limitations

    Single-segment representation learning is possible and can outperform methods based on agreement or disagreement among multiple audio segments.

    Source: §V · p. 7

Abstract

from arXiv · show

Inspired by the recent progress in self-supervised learning for computer vision that generates supervision using data augmentations, we explore a new general-purpose audio representation learning approach. We propose learning general-purpose audio representation from a single audio segment without expecting relationships between different time segments of audio samples. To implement this principle, we introduce Bootstrap Your Own Latent (BYOL) for Audio (BYOL-A, pronounced "viola"), an audio self-supervised learning method based on BYOL for learning general-purpose audio representation. Unlike most previous audio self-supervised learning methods that rely on agreement of vicinity audio segments or disagreement of remote ones, BYOL-A creates contrasts in an augmented audio segment pair derived from a single audio segment. With a combination of normalization and augmentation techniques, BYOL-A achieves state-of-the-art results in various downstream tasks. Extensive ablation studies also clarified the contribution of each component and their combinations.

I. INTRODUCTION

Previous audio methods often learn from relationships among segments at different times, but such relationships can be contradictory. BYOL-A instead learns general-purpose representations from one segment by creating augmented views focused on foreground sounds and texture details.

Motivation

  • Different-time segment relationships can be misleading because repetitive sounds may recur remotely while short events may occur unpredictably.These problems also affect contrastive and triplet-loss methods because their objectives compare multiple samples.Source: §I · p. 1

Method

  • The augmentation design targets foreground acoustic events and sound-texture details as complementary parts of a general-purpose representation.Foreground consistency is encouraged through random background variation, while pitch shifting and time stretching preserve useful details across variations.Source: §I · p. 2

Contribution

  • BYOL-A learns general-purpose audio representations from a single audio segment without expecting relationships between different time segments.This avoids the multiple-segment comparisons used by many previous audio methods.Source: §I · pp. 1–2

Method

  • Normalization, mixup, and random resize crop form the core augmentation strategy, with pre- and post-normalization supporting stable augmentation and statistical correction.The paper also reports extensive ablation studies examining the contribution of each augmentation block.Source: §I · p. 2
  • BYOL-A applies BYOL to audio by creating two augmented views from one segment and minimizing loss between their projected embeddings.The online network is updated by the loss, while the target network follows the online network through an exponential moving average.Source: Figure 1 · p. 1, §I · p. 2

II. RELATED WORK

Related audio methods commonly exploit temporal relationships or contrastive comparisons, while BYOL avoids negative samples and supports learning from a single input. BYOL-A also differs from prior mixup-based approaches in using mixup specifically to model audio content.

BYOL

  • BYOL avoids negative samples by minimizing the mean squared error between embeddings of augmented views from the same input.Its architecture and training algorithm are designed to avoid collapsed representations while claiming state-of-the-art performance.Source: A. Self-supervised learning schemes · p. 2

Mixup methods

  • Prior mixup-based methods such as DACL, i-MIX, and mix-back remain based on contrastive loss comparing positive and negative samples.The paper identifies this contrastive objective as a fundamental difference from BYOL-A.Source: A. Self-supervised learning schemes · p. 2

Audio representation learning

  • CPC, Jansen et al., TRILL, and COLA learn representations using relationships among segments cropped from different times.These methods use contrastive or triplet objectives to structure representations according to temporal proximity or clip membership.Source: A. Self-supervised learning schemes · p. 2

BYOL-A

  • The paper distinguishes BYOL-A from prior approaches by combining single-input learning with audio augmentations designed around foreground and content information.The supplied figure captions identify the BYOL-A system and its audio augmentation and random-resize-crop modules.Source: Figure 2 · p. 3, Figure 3 · p. 3, Figure 4 · p. 3

B. Bootstrap Your Own Latent (BYOL)

BYOL learns representations by matching augmented views of the same input without negative samples. An online network is trained against a slowly moving target network, with a predictor helping avoid collapse.

III. BYOL FOR AUDIO (BYOL-A)

BYOL-A adapts BYOL to learn general-purpose audio representations from a single segment rather than relationships among multiple time segments. Its audio augmentations target foreground events and content details, with normalization supporting stable processing.

III. BYOL FOR AUDIO (BYOL-A)

  • BYOL-A learns general-purpose audio representations from a single audio segment without expecting relationships between different time segments.Source: §III · p. 3
  • BYOL-A replaces BYOL’s image augmentation module with an audio module that creates contrasting augmented views from log-scaled mel-spectrograms.Source: §III · p. 3
  • The augmentation module contains four blocks, beginning with pre-normalization and duplicating the normalized input before mixup.Source: §III · p. 3
  • Mixup creates contrast for foreground acoustic-event representations, while Random Resize Crop approximates pitch shifting and time stretching for content details.Source: §III · pp. 3–4
  • Input normalization uses the training-sample mean µ and standard deviation σ, producing a standardized representation for subsequent processing.Source: A. Pre-Normalization · p. 4
  • Normalization stabilizes augmentation by reducing parameter sensitivity and statistical differences between training datasets.Source: A. Pre-Normalization · p. 4

B. Mixup for foreground acoustic event

BYOL-A’s mixup augmentation varies background content while preserving the main sound, encouraging foreground-event invariance. Random Resize Crop supplies complementary pitch and time variations for learning content details.

B. Mixup for foreground acoustic event

C. RRC for all content details

D. Post-Normalization for statistical drift adjustment

Post-normalization adjusts statistical drift introduced by earlier augmentations so BYOL-A’s final views approximate a standard normal distribution. The resulting representations are evaluated across six downstream audio tasks using frozen pretrained encoders.

D. Post-Normalization for statistical drift adjustment

  • Post-normalization adjusts augmentation-induced statistical drift so final BYOL-A views become approximately N(0, 1).It uses means and standard deviations calculated from batch samples.Source: D. Post-Normalization for statistical… · p. 4

IV. EXPERIMENTS

  • BYOL-A representations are evaluated on six audio downstream tasks with a linear model trained on frozen BYOL-A encoder embeddings.Source: §IV · p. 4
  • BYOL-A is pretrained on AudioSet, a large-scale dataset commonly used in previous studies.Source: §IV · p. 4
  • Pretraining and evaluation are repeated three times for full AudioSet and five times for 1/10 AudioSet or FSD50K, with results averaged.Source: A. Experimental Setup · p. 4
  • Audio is converted to 16,000-Hz log-mel spectrograms with 64-ms windows, 10-ms hops, and 64 mel bins.Source: 1) Audio data format and conversion… · p. 4
  • Pretraining uses randomly cropped 96-frame segments corresponding to 1,014 ms, while downstream segment lengths follow each dataset’s average duration.Source: 1) Audio data format and conversion… · pp. 4–5

2) Encoder network:

The encoder uses a compact CNN with a tunable representation dimension, and the study evaluates it across six downstream audio classification tasks.

Encoder architecture

  • BYOL-A uses a simple CNN based on a DCASE 2020 audio-captioning solution.The authors describe this as a realistic design choice because smaller networks are used in audio machine learning.Source: 2) Encoder network · p. 5
  • The representation embedding dimension is varied as a hyper-parameter and also sets the linear-layer sizes.This variation tests how representation size affects performance.Source: 2) Encoder network · p. 5
  • The projection and prediction heads use MLPs with a 4,096-unit linear layer and 256-dimensional output embeddings.They include batch normalization and ReLU, with Adam optimization at learning rate 0.0003.Source: 3) BYOL-A pretraining details · p. 5

Pretraining setup

  • Pretraining uses 1,963,807 unlabeled AudioSet samples, while ablations use a 1/10 subset containing 210,315 samples.The full AudioSet and subset settings support the main experiments and ablation studies, respectively.Source: 3) BYOL-A pretraining details · p. 5

Evaluation tasks

  • The downstream evaluation covers musical instrument, acoustic scene, speaker, language, and command classification tasks.The tested datasets are NSynth, UrbanSound8K, VoxCeleb1, VoxForge, Speech Commands V2, and SPCV2/12.Source: 4) Downstream tasks · p. 5
  • SPCV2/12 uses ten words plus silence and others, creating class imbalance and greater complexity than SPCV2.The setup has fewer classes but includes aggregated and background-noise-derived categories.Source: 4) Downstream tasks · p. 5

5) Linear evaluation details:

The study evaluates frozen BYOL-A encoders with linear classifiers and compares them against prior representation-learning methods under matched experimental settings.

Linear evaluation

  • Linear evaluation trains one supervised linear layer on frozen encoder embeddings extracted from downstream audio samples.Each evaluation runs ten times and reports averaged accuracy, with early stopping during 200-epoch maximum training.Source: 5) Linear evaluation details · p. 5

Comparison setup

  • COLA’ reproduces COLA with an added normalization module and the same encoder as BYOL-A, enabling comparison of single- and two-segment inputs.Its pretraining settings match BYOL-A except for a batch size of 1,024.Source: 6) Comparison with COLA [14] · p. 5
  • Table I compares TRILL, COLA, OpenL3, COALA, COLA’, and BYOL-A across the downstream tasks.BYOL-A and COLA’ are also tested with 512-, 1,024-, and 2,048-dimensional embeddings.Source: B. Experimental results: Comparison… · p. 5, Table I · p. 6

Results

  • BYOL-A with 2,048 dimensions outperforms the other methods in all tasks.The reported average is 77.8%, compared with 68.7% for COLA’ with 2,048 dimensions.Source: B. Experimental results: Comparison… · p. 5
  • BYOL-A with 512-dimensional embeddings also shows competitive performance, especially on the speech command tasks.This indicates that the strongest reported configuration is not the only competitive embedding size.Source: B. Experimental results: Comparison… · p. 6

C. Ablation study: Contribution of data augmentations

Ablations show that mixup and Random Resize Crop contribute differently, with their combination producing the strongest average performance among tested augmentation settings.

1) Contribution of mixup compared with Gaussian-noise

2) Contribution of mixup, RRC, and their combination

D. Ablation study: Contribution of normalization blocks

Normalization supports BYOL-A by stabilizing mixup-related computations and correcting augmentation-induced statistical drift, and both normalization blocks contribute to performance.

Normalization ablation

  • Removing pre-normalization degrades performance by −1.8 to −3.4, exceeding the −0.2 degradation from removing post-normalization.The ablation removes either block but retains the other to preserve the basic setup.Source: D. Ablation study: Contribution of… · p. 6
  • Pre-normalization stabilizes log-mixup-exp by making the log-mel spectrogram range constant.Without it, the mixup α sweet spot shifts, and α values down to 0.05 do not recover 72.3% performance.Source: D. Ablation study: Contribution of… · pp. 6–7
  • Normalization and augmentation together contribute to performance gain and recovery.The conclusion attributes BYOL-A’s overall result to the modules working together as one augmentation module.Source: D. Ablation study: Contribution of… · p. 7, §V · p. 7

APPENDIX

The appendix details the encoder and evaluates BYOL-A augmentation blocks, comparisons with COLA’+, and pretraining on FSD50K alongside AudioSet.

Encoder architecture

  • The appendix describes the encoder architecture and its fixed-dimensional representation outputs across 512-, 1,024-, and 2,048-dimensional settings.The corresponding parameter counts are 600,192, 1,649,792, and 5,321,856, respectively.Source: A. Details of encoder network · p. 7

Augmentation blocks

  • Mixup improved COLA’+ accuracy by 5.8, compared with 1.8 for Gaussian noise.COLA’+ extends COLA’ with augmentation blocks, whereas the original COLA does not use data augmentations.Source: B. Experiments on BYOL-A augmentation… · p. 7
  • Adding RRC to COLA’+Mixup improved performance by 1.5, versus an 8.4 improvement when RRC was applied to BYOL-A with Mixup only.The appendix attributes the smaller gain to resizing and cropping segments that were already randomly cropped.Source: B. Experiments on BYOL-A augmentation… · pp. 7–8
  • The BYOL-A augmentation module was effective with COLA’+, but more effective with BYOL-A’s single-segment input setting.This comparison is summarized across the augmentation-block experiments.Source: B. Experiments on BYOL-A augmentation… · p. 8

Pretraining datasets

Related papers

Loading 2103.06695v2…