PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep convolutional neural networks for predominant instrument recognition in polyphonic music

Yoonchang Han, Jaehun Kim, Kyogu Lee

arXiv:1605.09507v3cs.SDcs.CVcs.LGcs.NE

TL;DR

Predominant instrument recognition in real-world polyphonic music is difficult but useful for music search and other music-information-retrieval applications. The paper uses a ConvNet trained on single-labeled fixed-length excerpts, then aggregates sliding-window outputs to estimate multiple instruments in variable-length audio. On 10k excerpts covering 11 instruments, the proposed approach achieved micro-F1 0.602 and macro-F1 0.503, improving on other state-of-the-art algorithms.

  • Problem

    Polyphonic mixtures and real-world variation in timbre, quality, and playing style make automatic instrument recognition challenging, despite its utility for music search and other applications.

  • Method

    A ConvNet learns from fixed-length single-labeled excerpts, while sliding-window output aggregation estimates an arbitrary number of predominant instruments in variable-length recordings.

  • Results

    The proposed ConvNet outperformed previous state-of-the-art approaches on predominant instrument identification using Mel-spectrogram input without source separation.

  • Takeaways & Limitations

    The framework supports predominant instrument identification in real-world polyphonic music with multiple instruments estimated from variable-length clips.

Abstract

from arXiv · show

Identifying musical instruments in polyphonic music recordings is a challenging but important problem in the field of music information retrieval. It enables music search by instrument, helps recognize musical genres, or can make music transcription easier and more accurate. In this paper, we present a convolutional neural network framework for predominant instrument recognition in real-world polyphonic music. We train our network from fixed-length music excerpts with a single-labeled predominant instrument and estimate an arbitrary number of predominant instruments from an audio signal with a variable length. To obtain the audio-excerpt-wise result, we aggregate multiple outputs from sliding windows over the test audio. In doing so, we investigated two different aggregation methods: one takes the average for each instrument and the other takes the instrument-wise sum followed by normalization. In addition, we conducted extensive experiments on several important factors that affect the performance, including analysis window size, identification threshold, and activation functions for neural networks to find the optimal set of parameters. Using a dataset of 10k audio excerpts from 11 instruments for evaluation, we found that convolutional neural networks are more robust than conventional methods that exploit spectral features and source separation with support vector machines. Experimental results showed that the proposed convolutional network architecture obtained an F1 measure of 0.602 for micro and 0.503 for macro, respectively, achieving 19.6% and 16.4% in performance improvement compared with other state-of-the-art algorithms.

I. INTRODUCTION

The paper addresses predominant instrument recognition in professionally produced Western polyphonic music, where overlapping sounds and varying timbre, quality, and playing style complicate automatic identification. It proposes a ConvNet framework trained on single-labeled excerpts and aggregates sliding-window outputs to estimate multiple instruments in variable-length recordings.

  • Polyphonic mixtures and variation in timbre, quality, and playing style make automatic instrument identification difficult.
  • Instrument information supports music search, audio tagging, tailored equalization, recommendation, source separation, transcription, and genre identification.
  • The proposed ConvNet trains on single-labeled data while estimating an unknown number of instrument classes from multi-labeled target recordings.
  • Sliding-window ConvNet outputs are aggregated to identify predominant instruments in variable-length excerpts, addressing failures of conventional majority voting.
  • The study evaluates activation functions and other performance factors, including analysis window size, aggregation strategy, and identification threshold.

II. PROLIFERATION OF DEEP NEURAL NETWORKS IN MUSIC INFORMATION RETRIEVAL

Deep learning learns higher-level representations from raw inputs through stacked nonlinear layers, while ConvNets exploit local correlated patterns that can recur across time-frequency representations. Their application to music information retrieval has expanded, but prior instrument studies differed in data realism and label structure.

  • Traditional machine-learning systems typically require hand-crafted features, whereas deep learning automatically discovers higher-level representations from raw data.
  • ConvNets are suited to local groups of correlated values and have produced strong results across image processing and multimedia retrieval tasks.
  • The proposed architecture uses repeated double-convolution layers, max-pooling, global max-pooling, a fully connected layer, and 11 sigmoid outputs.
  • Music time-frequency representations contain instrument harmonics whose spectral characteristics resemble localized image patterns across time and frequency.
  • Earlier instrument-identification studies used isolated tones or non-predominant settings, whereas this research targets polyphonic music with single-label training and multi-label estimation.

III. SYSTEM ARCHITECTURE

The system preprocesses audio into normalized mono waveforms and applies a deep ConvNet designed for time-frequency inputs. Its architecture uses repeated convolution and pooling blocks, global max-pooling, and sigmoid outputs to support simultaneous instrument predictions.

  • Audio preprocessing: The preprocessing converts stereo audio to mono, downsamples it to 22,050 Hz, and normalizes the waveform by its maximum value.The resulting Nyquist frequency is 11,025 Hz, covering most instrument harmonics while removing higher-frequency noise.
  • Network architecture: The ConvNet follows deep AlexNet- and VGGNet-style designs with repeated convolutional layers followed by max-pooling.
  • Network architecture: The architecture is adapted to the input data using small receptive fields, fixed stride, padding, and increasing channel counts across convolutional layers.
  • Network architecture: After eight convolutional layers, global max-pooling feeds a fully connected layer; empirically, this outperformed global average pooling for the task.
  • Network architecture: Sigmoid outputs are used instead of softmax because multiple instruments may be present simultaneously.

C. Training Configuration

Training uses cross-entropy optimization, dropout regularization, and experiments across time resolutions and activation functions. Testing applies sliding windows and aggregates class-wise sigmoid outputs to estimate instruments in separate audio excerpts.

  • Training configuration: The network is optimized with categorical crossentropy using Adam with a 0.001 learning rate and minibatches of 128.
  • Training configuration: Dropout is applied after pooling layers at rate 0.25 and after the fully connected layer at rate 0.5 to reduce overfitting.
  • Analysis resolution: Training evaluates 3.0, 1.5, 1.0, and 0.5-second analysis windows using divided fixed-length audio excerpts.
  • Activation function: The activation-function experiment compares tanh, ReLU, LReLU, and PReLU, with LReLU tested at α = 0.33 and α = 0.01.
  • Testing and aggregation: Separate test audio is analyzed with half-window-hop sliding windows, whose class-wise sigmoid outputs are summed to obtain instrument activations.

A. IRMAS Dataset

The IRMAS dataset pairs single-label training excerpts with variable-length, multi-label test audio for predominant instrument recognition. Testing uses sliding-window outputs aggregated by instrument to support arbitrary numbers of predicted instruments.

  • Dataset composition: Training contains 3-second excerpts from more than 2,000 recordings, while testing uses separate 5–20-second audio files.The training set has 6,705 files; the test set has 2,874 files.
  • Label structure: Training excerpts contain one continuously predominant instrument, whereas test excerpts may contain one or more predominant target instruments.Thus, training labels correspond one-to-one with files, while test annotations are multi-label.
  • Testing configuration: The system analyzes variable-length test audio with overlapping short-time windows to obtain local instrument information.The testing audio is longer than the fixed-shape training input.
  • Aggregation strategies: S1 averages instrument-wise sigmoid outputs, while S2 sums them and normalizes by the maximum class activation.S1 may return no detected instrument; S2 emphasizes relative strength among instruments.
  • Decision rule: Majority voting is avoided because overlapping, weaker accompaniment instruments could otherwise be disregarded.The method is designed to identify an arbitrary number of predominant instruments.
  • Decision rule: All instruments whose aggregated value exceeds the identification threshold are treated as predominant instruments.Higher thresholds favor precision, whereas lower thresholds favor recall.

C. Performance Evaluation

Performance is evaluated with precision, recall, and F1 measures using both micro and macro averaging. This accounts for unequal annotation counts across the 11 instrument classes.

  • Evaluation metrics: Precision, recall, and F1 are used to evaluate the instrument recognition system.F1 is the harmonic mean of precision and recall.
  • Averaging schemes: Micro averages compute metrics globally across classes, giving more weight to instruments with more appearances.This contrasts with macro averaging, which weights labels equally.
  • Averaging schemes: Macro averages compute each label’s metric separately and then take the unweighted average across labels.The distinction is motivated by unequal numbers of annotations for the 11 instruments.

V. RESULTS

The proposed ConvNet is evaluated against existing algorithms and across activation functions, analysis windows, aggregation strategies, and thresholds. It performs best with the stated default configuration and outperforms prior methods on IRMAS.

  • Experimental settings: The default experiment settings were LReLU (α = 0.33), a 1-second analysis window, S2 aggregation, and a 0.50 threshold.These settings showed the best performance where applicable.
  • Comparison to Existing Algorithms: 0.602 micro F1 and 0.503 macro F1 were achieved by the proposed ConvNet on the IRMAS dataset.The compared methods used hand-crafted timbral features with SVMs, with Bosch et al. adding source separation.
  • Comparison to Existing Algorithms: The ConvNet outperformed existing algorithms on both micro and macro F1 measures.Fuhrmann and Herrera’s method had the highest precision but recall around 0.25, producing a low F1 measure.
  • Effect of Activation Function: LReLU with α = 0.33 produced the best identification performance among the tested activation functions.LReLU with α = 0.01 performed similarly to ReLU, while PReLU matched ReLU rather than improving on it.
  • Effect of Activation Function: Leaky ReLU outperformed normal ReLU, which the authors attribute to retaining a nonzero negative-region gradient.The paper connects this behavior with preventing initially inactive units from remaining inactive.

C. Effect of Analysis Window Size

Analysis window size strongly affects instrument-identification performance: 3.0 seconds performs poorly, 0.5 seconds also hurts performance, and 1.0 second is optimal.

  • C. Effect of Analysis Window Size: 3.0-s windows clearly perform worse than shorter windows across identification thresholds.The experiment compares 3.0, 1.5, 1.0, and 0.5 s analysis windows using micro and macro F1.
  • C. Effect of Analysis Window Size: 1.0 s is the optimal analysis window size for the task.Shorter windows improve temporal resolution, but 0.5 s is too short for identifying instruments.
  • C. Effect of Analysis Window Size: 0.5-s windows decrease overall performance again after the improvement obtained by shortening longer analysis windows.This indicates that the shortest tested window is too brief for reliable instrument identification.
  • C. Effect of Analysis Window Size: 0.5 is the most appropriate identification threshold because it gives the best macro F1 performance.Higher thresholds improve precision but reduce recall, whereas lower thresholds improve recall but reduce precision.

E. Effect of Aggregation Strategy

Aggregation and instrument characteristics affect identification unevenly. The class-wise sum followed by normalization generally outperforms averaging, while class performance varies with data availability and spectral distinctiveness.

  • E. Effect of Aggregation Strategy: S2 shows better overall identification performance than S1, with a more notable advantage for macro F1 than micro F1.S2 sums outputs class-wise and then normalizes, whereas S1 takes class-wise means.
  • E. Effect of Aggregation Strategy: Class-wise sum followed by normalization is better than class-wise mean aggregation for predominant instrument identification.Audio-excerpt-wise normalization may reduce the effect of substantial quality differences between training and testing recordings.
  • F. Analysis of Instrument-Wise Identification Performance: Voice achieves an F1 measure of about 0.90, whereas cello and clarinet achieve around 0.20.Identification performance varies substantially by instrument regardless of parameter setting.
  • F. Analysis of Instrument-Wise Identification Performance: 338 cello and 505 clarinet training excerpts were available, compared with 778 for voice, potentially affecting class-wise performance.The authors believe more cello and clarinet training data could improve identification for those instruments.
  • F. Analysis of Instrument-Wise Identification Performance: 111 cello and 62 clarinet test excerpts make those evaluation results less reliable and stable than results for other instruments.The dataset contains 1,044 human-voice test excerpts for comparison.
  • F. Analysis of Instrument-Wise Identification Performance: Voice performance is likely high because human voice has distinctive inharmonic spectral characteristics and natural vibrato.Other instruments in the experiment generally produce clearer harmonic patterns.
  • F. Analysis of Instrument-Wise Identification Performance: A 3-s window considerably decreases performance for flute, acoustic guitar, electric guitar, and violin, while a 0.20 threshold harms several lead or wind instruments.The 0.20 threshold particularly reduces performance for flute, saxophone, trumpet, and violin but slightly improves electric guitar, organ, and piano.

G. Qualitative Analysis with Visualization Methods

The qualitative analysis examines how the ConvNet organizes instrument representations across layers using t-SNE and deconvolution. Increasingly high-level activations separate instrument classes, while individual units reveal spectrogram patterns.

  • G. Qualitative Analysis with Visualization Methods: t-SNE visualizes clustering at the ends of four convolutional blocks, the hidden dense layer, and the final sigmoid output.The analysis compares training samples with validation samples.
  • G. Qualitative Analysis with Visualization Methods: Higher encoding levels group same-instrument samples more tightly and separate different instrument groups farther apart, especially on training data.Validation data show a similar clustering tendency, though less clearly.
  • G. Qualitative Analysis with Visualization Methods: Deconvolution reverses the operations leading to a target unit to generate a visually inspectable representation of that unit’s learned functionality.The method is applied to units throughout the proposed ConvNet using input mel-spectrograms.
  • G. Qualitative Analysis with Visualization Methods: Most first-layer units extract vertical, horizontal, and diagonal edges from the input spectrogram.These patterns correspond to lower-level spectrogram features learned by the network.
  • G. Qualitative Analysis with Visualization Methods: First-layer units detect harmonic components through horizontal spectrogram edges, while higher-layer units exhibit different behavior for signals labeled voice versus acoustic guitar.Both signals contain voice and acoustic guitar, but their predominant-instrument labels differ.
  • G. Qualitative Analysis with Visualization Methods: Some learned units respond to regions containing strumming acoustic-guitar sound in the mel-spectrogram.This observation links unit activation patterns to an instrument-specific acoustic structure.

VI. CONCLUSION

The paper applies ConvNets to predominant instrument recognition in real-world polyphonic music, using fixed-length single-labeled training data and variable-length audio aggregation. Experiments identify effective activation, aggregation, window, and threshold choices, while visualizations show progressively more instrument-specific representations.

  • VI. CONCLUSION: The ConvNet identifies an arbitrary number of predominant instruments in variable-length music clips after training on fixed-length, single-labeled excerpts.Multiple outputs are aggregated to produce an audio-excerpt-wise result.
  • VI. CONCLUSION: The proposed ConvNet outperformed previous state-of-the-art approaches on predominant instrument identification in the IRMAS dataset without source separation preprocessing.Mel-spectrograms were used as the network input.
  • VI. CONCLUSION: LReLU performed better than normal ReLU, especially with the very leaky setting α = 0.33, while Tanh performed worse than other rectifier functions.PReLU matched ReLU for this task.
  • VI. CONCLUSION: S2 was the better aggregation method; a 1.0 s analysis window and identification threshold 0.5 produced the best reported settings.Shorter windows improved temporal resolution, but 0.5 s was too short for accurate identification; threshold changes traded precision against recall.
  • VI. CONCLUSION: t-SNE and deconvolution visualizations showed progressively clearer representations, with higher layers combining lower-level edges into spectral characteristics of instruments.Lower layers tended to capture horizontal and vertical edges.
  • VI. CONCLUSION: The study indicates that neural-network advances from image processing can transfer to audio processing, while music-specific overlap, repetition, continuity, and variable duration remain important differences.The authors suggest musical knowledge and adaptive instrument-specific thresholding as directions for further investigation.
Loading 1605.09507v3…