PaperlayerView exact revision on arXiv

Source-linked AI summary

Harmonica: Accurate and Lightweight Instrument-Agnostic Music Transcription

Longshen Ou, Héctor Martel, Joe Hennessy-Priest, Taemin Cho

arXiv:2609.04640v1cs.SD

TL;DR

Instrument-agnostic AMT must generalize across diverse pitched instruments while remaining computationally practical, but existing approaches leave this accuracy-efficiency trade-off unresolved. Harmonica uses multi-depth harmonic convolution in efficient convolutional models and achieves the best overall performance at each evaluated scale, including state-of-the-art x-large accuracy and a highly efficient nano variant.

  • Problem

    Existing AMT approaches are often instrument-specific, while reliable generalization across diverse pitched instruments and efficient scaling remain limited.

  • Method

    Harmonica is a family of convolution-based, instrument-agnostic transcription models built around multi-depth harmonic convolution and evaluated under a common multi-source protocol.

  • Results

    At each model scale, Harmonica achieves the best overall performance among evaluated models, with x-large state-of-the-art accuracy and the medium variant faster than every baseline.

  • Takeaways & Limitations

    Multi-depth harmonic convolution provides an effective, parameter-efficient way to exploit harmonic information across computational budgets for practical instrument-agnostic AMT.

Abstract

from arXiv · show

This paper introduces Harmonica, a family of instrument-agnostic music transcription models built around multi-depth harmonic convolution. At each model scale, Harmonica achieves the best performance among the evaluated models: the x-large model attains state-of-the-art performance in instrument-agnostic transcription, while the medium variant offers competitive accuracy with faster inference than all baselines. Pushing the limit of computational efficiency, the nano variant has only 26.3K parameters and runs at 1,622.5 times real time, yet achieves a frame F1 of 0.796 on the development set, 14.6 percentage points higher than Basic Pitch. We further demonstrate that multi-depth harmonic convolution effectively exploits harmonic information to benefit transcription performance through comparative experiments with existing harmonic aggregation methods, including harmonic stacking, harmonic attention, single-depth harmonic convolution, and the HD-Conv layer.

1. INTRODUCTION

The paper targets instrument-agnostic transcription across diverse pitched instruments and asks whether accuracy and efficiency can scale together. Harmonica answers with multi-depth harmonic convolution and strong performance across model sizes.

  • Instrument-specific AMT systems have limited applicability to recordings containing diverse musical instruments.
  • Instrument-agnostic transcription remains underexplored relative to multi-instrument transcription, leaving reliable individual-instrument generalization unclear.
  • Increasingly large AMT models improve accuracy but constrain deployment on devices with limited computation and memory.
  • Harmonica benchmarks recent AMT models under a common multi-source, instrument-agnostic protocol and uses multi-depth harmonic convolutions across computational budgets.
  • 28.5% fewer harmonic-layer parameters matches HD-Conv performance in the parameter-efficient formulation.
  • At each model scale, Harmonica achieves the best overall performance among evaluated models, with x-large state-of-the-art accuracy and faster or more accurate smaller variants.
  • The benchmark covers eight test sets spanning diverse instrument properties under a common multi-source, instrument-agnostic setting.

2. RELATED WORKS

Prior AMT work includes instrument-agnostic, sequence-to-sequence, frame-level, and harmonic-aggregation approaches. The paper identifies unresolved questions about efficient generalization and the most effective harmonic aggregation strategy.

  • Basic Pitch and HFSFormer address instrument-agnostic transcription, while much recent work focuses on single instruments or full multi-instrument mixtures.
  • Sequence-to-sequence AMT models learn acoustic-to-symbolic mappings without explicit musical priors, demanding large models and training sets.
  • Frame-level models avoid heavy autoregressive decoders but can still require substantial capacity to learn musical structure.
  • Harmonic aggregation improves frame-level efficiency through methods including HD-Conv, 3-D harmonic layers, and attention-based aggregation.
  • The most effective harmonic aggregation strategy for instrument-agnostic transcription remains unclear because existing convolutional approaches often aggregate harmonic information only once.

3. METHOD

Harmonica converts CQT audio into frame-level onset, sustain, and offset predictions, then uses harmonic convolution throughout a deep convolutional trunk. Its design aligns frequency with pitch and aggregates harmonic information at multiple abstraction levels.

  • 3.1. Problem Formulation: Instrument-agnostic transcription recovers onset times, offset times, and MIDI pitches for an unknown number of potentially overlapping notes.
  • 3.1. Problem Formulation: A learned network predicts frame-level onset, sustain, and offset activation matrices from a time-frequency representation.
  • 3.1. Problem Formulation: Decoding thresholds onset peaks and tracks sustain until it falls below a separate threshold, while offset activation provides auxiliary supervision only.
  • 3.2. Input Representation: CQT input provides constant semitone resolution across the pitch range and aligns the input frequency axis with the output pitch axis.
  • 3.3. Model: The model processes 16 kHz CQT spectrograms through a shallow ResNet frontend before downsampling frequency to one bin per semitone.
  • 3.3. Model: Each trunk block pairs harmonic convolution with a ResNet block, so harmonic aggregation occurs at multiple levels of abstraction before three prediction heads.
  • 3.3. Model: Harmonic convolution evaluates harmonic and subharmonic energy to refine whether a frequency bin represents a fundamental.
  • 3.3. Model: HD-Conv uses differently dilated branches whose outputs are summed, but redundantly parameterizes the center bin in every branch.

4. EXPERIMENTS

The experiments benchmark instrument-agnostic transcription under a common protocol and evaluate Harmonica across model scales, instrument families, and harmonic aggregation strategies. Harmonica delivers strong accuracy-efficiency trade-offs, while multi-depth harmonic convolution improves over alternative aggregators.

  • 4.1. Experiment Settings: The evaluation uses eight datasets, multi-source training, eleven recent transcription baselines, and note- and frame-level F1 metrics.The protocol includes instrument-agnostic track-wise audio–MIDI evaluation and a held-out development set.
  • 4.2. Main Result: At every scale, Harmonica achieves the best overall performance among the evaluated models.The x-large model ranks first in frame F1 on all in-domain test sets and in OnP F1 except URMP-stem.
  • 4.2. Main Result: 848.5× real time: the medium model performs comparably to the best baseline in every in-domain column while running faster than all baselines.The nano model uses 26.3K parameters, runs at 1,622.5× real time, and outperforms Basic Pitch on every test set except URMP.
  • 4.2. Main Result: The x-large model is strongest across every instrument family, while PerceiverTF and HFSFormer degrade particularly on chromatic percussion and synth pad.The family-level breakdown indicates more even scores for Harmonica across diverse pitched instruments.

5. CONCLUSION

The paper benchmarks instrument-agnostic transcription and introduces Harmonica, an efficient convolution-based model family that scales across computational budgets. Its multi-depth harmonic convolution provides a parameter-efficient way to exploit harmonic information, and the system has been deployed in BandLab.

  • 5. CONCLUSION: Harmonica consistently outperforms prior models at comparable computational scales under a common instrument-agnostic protocol.The benchmark shows that existing approaches vary substantially in their ability to generalize across instruments.
  • 5. CONCLUSION: Multi-depth harmonic convolution effectively and parameter-efficiently exploits harmonic information for music transcription.The conclusion identifies efficient, instrument-agnostic architectures as promising for practical AMT.
  • 5. CONCLUSION: Harmonica is deployed as BandLab’s Audio-to-MIDI service to support accessible music creation.

A. EFFICIENCY OF ON-DEVICE DEPLOYMENT

The medium Harmonica model is evaluated on an iPhone 15 Pro Max to measure practical mobile inference speed and memory use. It runs faster than Basic Pitch on mobile hardware despite having roughly 40 times more parameters, while using less peak memory than the cited Basic Pitch figure.

  • CPU deployment: 7.0× real time: the medium model transcribes 156 seconds of audio in 22.16 seconds on the iPhone CPU.CPU peak memory is 382.6 MB.
  • GPU deployment: 32.1× real time: LiteRT on the iPhone GPU reduces inference time to 4.86 seconds for the 156-second recording.GPU peak memory is 445.0 MB.
  • Comparison: The medium model runs faster than Basic Pitch on mobile hardware despite having roughly 40 times more parameters.The comparison uses the cited Basic Pitch paper’s long-audio results on a 2017 MacBook Pro.

B. DATASET INFORMATION

The benchmark combines diverse pitched-instrument datasets with a held-out development set for monitoring and threshold selection.

  • Piano datasets: MAESTRO provides about 198.6 hours of aligned piano performance recordings for training and evaluation.
  • Piano datasets: MAPS contributes synthesized and recorded piano sounds, with two recorded subsets used as out-of-domain test sets.
  • Guitar datasets: GuitarSet contains acoustic-guitar improvisations from six experienced players across varied keys, tempos, and genres.
  • Guitar datasets: GAPS supplies aligned classical-guitar audio-score pairs, while GOAT includes substantial distorted electric-guitar material and is the benchmark’s most difficult dataset.
  • Multi-instrument datasets: URMP covers 13 classical instruments across string, brass, and woodwind families, and Slakh spans 11 pitched General MIDI families in synthesized pop tracks.
  • Development set: The held-out development set contains 98 songs totaling 4.33 hours and supports checkpoint selection and threshold sweeping without data leakage.

C. FULL COMPARISON TABLE

The full comparison tables include OnPOff F1, a metric omitted from the main benchmark table because of space limitations.

  • OnPOff F1 is reported in Tables 4 and 5 rather than the space-limited benchmark Table 1.

D. IMPLEMENTATION DETAILS

Implementation uses model-specific threshold sweeps and chunked song-level inference to manage prediction quality and memory use.

  • Threshold Sweep: Thresholds for onset and frame predictions are swept from 0.1 to 0.85 in 0.05 increments for models requiring tuning.
  • Batched Chunk Inference: Full-song inference is divided into overlapping chunks because processing long pieces at once exceeds available memory.
  • Batched Chunk Inference: Each chunk uses half-length overlap, and only its central half contributes to the concatenated song-level prediction.

E. PROBING EXPERIMENT

The probing experiment uses frozen intermediate representations to track how note information becomes linearly decodable across Harmonica’s layers.

  • Probe design: A linear probe is trained on each frozen internal layer to predict the sustain matrix from its feature map.
  • Probe design: Probe accuracy measures how linearly decodable note information is at each stage and how far representations have been refined toward explicit notes.
  • Layer roles: The frontend recovers substantial note information from the CQT but leaves false-positive energy at harmonic positions.
  • Layer roles: Harmonic blocks concentrate energy on fundamentals and suppress overtone responses, improving precision.
  • Layer roles: ResNet blocks denoise and sharpen activations, raising confidence in latent notes that cross the detection threshold and improve recall.
  • Layer roles: FGLSTM heads refine temporal structure and complete confidence sharpening.
Loading 2609.04640v1…