PaperlayerView exact revision on arXiv

Source-linked AI summary

JASPER: Joint Audio and Speech Pre-trained Encoder Representations

Geeth George, Ameenudeen P E, Hrishikesh H Pillai, Sriram Ganapathy

arXiv:2609.27260v1eess.AS

TL;DR

Speech and audio SSL have developed along separate lines, with different encoder designs limiting cross-domain applicability. JASPER combines temporal and spectral prediction in a shared waveform encoder, and reports stronger performance across diverse tasks, including a best overall average accuracy of 81.74% in clip-level classification.

  • Problem

    Speech and audio encoders use different design choices, and frame-level tokenizers and patch-based models have limitations on environmental-audio and temporally precise speech tasks, respectively.

    Source: §I · p. 1

  • Method

    JASPER jointly predicts temporal pseudo-labels and frequency-localized spectral targets from long audio contexts using a unified raw-waveform encoder.

    Source: §I · pp. 1–2

  • Results

    JASPER consistently outperforms established baselines across frame-level and utterance-level tasks, with a best overall average accuracy of 81.74% in clip-level classification.

    Source: 1) Clip-level Classification · p. 5, §VII · p. 6

  • Takeaways & Limitations

    The results highlight unified spectro-temporal modeling for capturing complex acoustic and semantic information required for large audio language models.

    Source: §VII · p. 6

Abstract

from arXiv · show

Self-supervised learning (SSL) for speech and audio has largely progressed along separate tracks: speech models emphasise time-domain prediction, whereas audio representation learning has focused on time-frequency patterns. This separation creates a compatibility gap, limiting cross-domain generalization. In this work, we introduce JASPER, Joint Audio and Speech Pre-trained Encoder Representations, a framework that augments speech-pretrained models with time-frequency objectives. Specifically, JASPER performs masked prediction of temporal and spectral targets over long audio segments, enabling spectro-temporal representation learning of speech and audio signals. The proposed method consistently outperforms multiple baselines and existing speech/audio encoders on diverse speech, audio, and music tasks, demonstrating the effectiveness of unified spectro-temporal modeling.

I. INTRODUCTION

Speech and audio encoders often specialize in different input modalities, limiting their cross-domain applicability. JASPER combines temporal and spectral prediction in a shared encoder and is evaluated across speech, audio, music, and audio-language-model tasks.

I. INTRODUCTION

  • Frame-level speech tokenizers can be less effective for environmental sounds, while patch-based audio models can miss the precise temporal information needed for speech tasks.These differences limit the cross-domain applicability of models designed for their respective domains.Source: §I · p. 1
  • JASPER jointly predicts temporal targets and spectral patch targets over long audio contexts with a shared Transformer encoder.Its spectral targets are time-aligned patches derived from log-Mel representations, while inference uses a raw-waveform encoder.Source: §I · pp. 1–2
  • The evaluation framework combines frozen-representation linear evaluations, encoder-centric X-ARES tests, and deployment as an acoustic front end for XARES-LLM.Source: §I · p. 1
  • Contiguous latent segments are masked to encourage the encoder to infer missing acoustic content from broader temporal context.Source: §I · p. 1
  • JASPER is presented as a universal audio encoder for broad speech, audio, music, and audio-LLM evaluations, spanning discriminative and generative audio understanding.Source: §I · pp. 1–2

II. JASPER ENCODER FRAMEWORK

JASPER processes raw speech or audio with a convolutional feature encoder, then groups the resulting temporal representations into non-overlapping windows. The framework is presented as a unified encoder for speech and general audio.

II. JASPER ENCODER FRAMEWORK

  • JASPER’s framework is a self-supervised audio encoder designed to learn unified representations for speech and general audio signals.Source: §II · p. 2

A. Input Pre-Processing

  • A convolutional feature encoder processes raw audio sampled at 16 kHz with an overall stride of 320, producing latent temporal representations at 50 Hz.Source: A. Input Pre-Processing · p. 2

B. Temporal Segmentation

  • The latent sequence is divided into non-overlapping windows of P frames, corresponding to segments of duration τ seconds, with P = 50τ.Each window groups consecutive latent vectors.Source: B. Temporal Segmentation · p. 2

C. Long-Context masking

JASPER masks latent windows, optionally extending masking to a subsequent window, and encodes the sequence with temporal positional information. Its temporal objective predicts K-means pseudo-labels only for masked frames.

C. Long-Context masking

  • Each latent window is masked independently with probability p, and a selected window’s successor is also masked with probability p′; experiments use p = 0.6 and p′ = 0.2.Source: C. Long-Context masking · p. 2

D. Transformer Encoding of Temporal Frames

E. Temporal Targets and Loss Function

  • K-means cluster assignments provide discrete frame-level pseudo-labels, and the temporal loss trains the Transformer to predict them only at masked frames.The labels correspond to individual frames within each window.Source: E. Temporal Targets and Loss Function · pp. 2–3

F. Spectral Targets and Loss

JASPER constructs frequency-localized spectral targets from log-Mel representations and predicts them from temporal segment representations. Its spectral loss is computed over masked windows.

F. Spectral Targets and Loss

  • Log-Mel spectrograms are divided into non-overlapping segments, each capturing the spectral structure of a temporal window of duration τ.The segments are further split into non-overlapping square patches.Source: F. Spectral Targets and Loss · p. 3
  • Flattened spectral patches are quantized with a K-means quantizer to produce spectral targets.The number of spectral clusters is denoted by Vs.Source: F. Spectral Targets and Loss · p. 3
  • Mean-pooled transformer embeddings for each temporal segment are passed to independent feed-forward heads to predict spectral cluster labels.The spectral loss is computed over masked windows.Source: F. Spectral Targets and Loss · p. 3

G. Combined Training Objective

JASPER combines its temporal and spectral losses in an overall training objective. The objective is their sum.

G. Combined Training Objective

  • The overall training objective combines the temporal loss Lt and spectral loss Ls.The passage describes the objective as the sum of the two losses.Source: G. Combined Training Objective · p. 3
  • The objective includes both temporal and spectral prediction losses.These are the two components named in the overall training objective.Source: G. Combined Training Objective · p. 3
  • JASPER's overall training objective is the sum of its spectral and temporal losses.The passage identifies the combined objective as the sum of Ls and Lt.Source: G. Combined Training Objective · p. 3

III. IMPLEMENTATION DETAILS

JASPER uses an 8-second audio input and a convolutional feature encoder followed by a 12-layer transformer. Its spectral setup uses 50 windows and eight vertical patches per segment.

III. IMPLEMENTATION DETAILS

  • Audio inputs are truncated or zero-padded to 8 seconds before encoding.The temporal segment duration τ is 0.16 seconds, chosen to match syllable-length units.Source: A. Model Architecture and Targets · p. 3
  • The waveform passes through a convolutional feature encoder and a 12-layer transformer with 12 attention heads and dimension D′ = 768.The convolutional encoder dimension is Dt = 512.Source: A. Model Architecture and Targets · p. 3
  • The spectral configuration uses F = 800 and Q = 16, yielding 50 non-overlapping windows with eight vertical patches per segment.Self-supervised target construction uses Vt = 500 and Vs = 500.Source: A. Model Architecture and Targets · p. 3

B. Pre-training Setup and Optimization

JASPER is pretrained on equal amounts of speech and audio in 200-hour and 2000-hour regimes. The evaluation setup combines frozen-representation downstream tests with X-ARES encoder-centric and audio-LLM pathways.

B. Pre-training Setup and Optimization

  • Pretraining uses equal amounts of speech and audio from LibriSpeech and AudioSet in 200-hour and 2000-hour regimes.The encoder is initialized with temporal-only targets, then trained with joint temporal and spectral objectives for 1M steps.Source: B. Pre-training Setup and Optimization · p. 3

IV. EVALUATION SETUP

  • SUPERB-style linear evaluation keeps the pretrained encoder frozen and trains only layer-aggregation weights and a lightweight task-specific prediction head.The evaluation covers speech, environmental sound, and music tasks, with classification accuracy, PER, WER, and CER among the reported metrics.Source: §IV · p. 3
  • The cited tables cover matched-data pretraining comparisons, 2000-hour baseline comparisons, phoneme recognition and ASR, and X-ARES and Track A/B evaluations.The supplied table captions identify these comparison scopes but do not provide result values.Source: Table I · p. 4, Table II · p. 4, Table III · p. 4
  • X-ARES evaluation uses MLP linear probing and non-parametric k-NN, while XARES-LLM routes frozen encoder embeddings through a language-model pipeline.The X-ARES MLP setup freezes the encoder and task-specific components and trains only a single linear layer; the LALM pipeline uses SmolLM2-135M.Source: §IV · pp. 3–4

A. Track A - Classification Tasks:

The two tracks distinguish utterance-level classification from higher-level semantic understanding and use task-appropriate metrics.

A. Track A - Classification Tasks:

B. Track B - Understanding Tasks:

A. Linear Evaluation

JASPER performs across clip- and frame-level tasks and encoder-centric and audio-language-model evaluations, with reported results spanning speech, audio, and music settings.

1) Clip-level Classification:

  • 81.74% overall average accuracy; JASPER outperforms spectral and temporal baselines on most speech and audio tasks in both 200-hour and 2000-hour regimes.Source: 1) Clip-level Classification · p. 5

2) Frame-level Sequence Prediction:

  • Frozen JASPER representations retain detailed speech-level information on frame-level sequence tasks despite added spectral prediction and audio-domain objectives.Source: 2) Frame-level Sequence Prediction · p. 5

B. Encoder-Centric Benchmark Evaluation (X-ARES)

  • JASPER performs best with parametric MLP probing and, in k-NN evaluation, outperforms HuBERT while its representations cluster by semantic similarity without classifier training.Source: B. Encoder-Centric Benchmark… · pp. 5–6

C. XARES-LLM Evaluation

  • Track A averages 0.66, surpassing Whisper with 2k versus 680k pre-training hours; Track B averages 0.50, leading WavLM and HuBERT.Source: C. XARES-LLM Evaluation · p. 6

VI. ABLATION STUDIES

The ablations report gains from increasing pre-training data and identify a loss-weight setting that balances spectral and temporal objectives; the conclusion summarizes JASPER’s cross-task results.

A. Scaling Improvements

  • Increasing pre-training from 200 to 2000 hours consistently improves performance across speech, environmental audio, and music tasks, including when JASPER is an audio-language-model front end.Source: A. Scaling Improvements · p. 6

B. Effect of Loss Weight λ

VII. CONCLUSION

  • JASPER’s continuous representations consistently outperform established baselines across frame- and utterance-level tasks, with improvements achieved using less training data than several baselines.Source: §VII · p. 6
Loading 2609.27260v1…