Source-linked AI summary
Finetuning Strategies for Querying Sounds by Vocal Imitation
Aditya Bhattacharjee, Christos Plachouras, Sungkyun Chang, Emmanouil Benetos
TL;DR
Query by Vocal Imitation must bridge variable human vocalizations and acoustically diverse sound classes to retrieve matching audio. This report compares frozen-encoder contrastive learning with MobileNetV3 contrastive-triplet learning, with the latter winning the AES AIMLA 2025 challenge and both strategies improving retrieval over baselines.
Problem
Query by Vocal Imitation requires representations that bridge vocal and non-vocal audio despite variable vocalizations and diverse sound classes.
Method
The report compares supervised contrastive learning with a frozen CED encoder against shared-MobileNetV3 contrastive-triplet learning using semi-hard negatives.
Results
Submission #2 won the AES AIMLA 2025 challenge, while both submissions yielded significant retrieval gains over baseline systems.
Takeaways & Limitations
Transfer learning and structured negative sampling are both valuable strategies for improving retrieval performance in QbVI systems.
Abstract
from arXiv · showhide
This technical report describes our winning submission to the AES AIMLA 2025 Challenge on querying sound effects by vocal imitation. We investigate two complementary fine-tuning strategies: contrastive learning with a frozen, pretrained CED encoder, and joint contrastive-triplet learning with semi-hard negatives using a MobileNetV3 encoder. This report has been updated for posterity to include details released after the challenge.
I. INTRODUCTION
Query by Vocal Imitation retrieves sound effects from vocal references, requiring representations that bridge vocal and non-vocal audio despite varied imitations and acoustic diversity. This work explores two neural submission strategies for the AES-AIMLA 2025 Challenge: contrastive fine-tuning of a frozen CED encoder and a MobileNetV3-based approach with triplet regularization.
- Task motivation: QbVI retrieves audio samples from a reference collection using a user-provided vocal imitation.The task must accommodate variability in human vocalizations and acoustic diversity across sound classes.
- Task motivation: Effective QbVI systems must bridge the modality gap between vocal and non-vocal audio while remaining robust to diverse imitation styles.This challenge motivates learning representations rather than relying only on conventional similarity measures.
- Related work: Earlier QbVI methods used autoencoder bottleneck features or semi-Siamese convolutional architectures with contrastive loss, often paired with traditional classifiers or similarity measures.Examples of similarity measures include dynamic time warping and cosine distance.
- Related work: Greif et al. established a strong neural QbVI baseline using a dual-tower MobileNetV3, SimCLR-style contrastive objectives, and extensive augmentation.Their audio embeddings were pretrained on large-scale datasets such as AudioSet.
- Contributions: This work investigates two complementary AES-AIMLA 2025 submissions: contrastive fine-tuning of a frozen CED encoder and a MobileNetV3-based setup integrating triplet regularization.The CED encoder was originally trained for audio tagging through knowledge distillation, while the second strategy builds on Greif et al.’s MobileNetV3 setup.
II. DATASETS
The study trains on matched VimSketch reference–imitation pairs and class-labeled VocalSketch imitations, using the latter as structured negatives for triplet learning. Both submissions apply independent online augmentation to query and reference audio, while validation uses the official qvim-dev set with MRR and NDCG.
- Datasets: VimSketch provides supervised reference–imitation pairs with known correspondence for the contrastive learning objective.
- Datasets: VocalSketch contains unmatched, sound-class-labeled imitations used as structured triplet negatives, prioritizing same-class samples and otherwise sampling randomly.
- Datasets: The first submission uses only VimSketch, whereas the second combines both datasets in a hybrid contrastive–triplet training regime.
- Augmentation: Online augmentation independently transforms query and reference audio during training to improve robustness and generalization across vocal styles, recording conditions, and intraclass variation.Transformations include time shifting, gain and pitch adjustment, time stretching, additive Gaussian noise, frame-level corruption, and frequency or time masking.
- Evaluation: Validation uses the official qvim-dev set from the QVIM 2025 Challenge and reports Mean Reciprocal Rank (MRR) and Normalized Discounted Cumulative Gain (NDCG).
III. SUBMISSION #1 · A. Architecture and Training Setup
Submission #1 uses a pretrained CED-base audio encoder with a lightweight projection head for retrieval. It trains shared query and reference branches using supervised contrastive learning, in-batch negatives, and symmetric online augmentation.
- A. Architecture and Training Setup: CED is a ViT-based audio encoder trained for AudioSet tagging with consistent ensemble distillation.It performed strongly on the HEAR 2021 benchmark, particularly for vocal imitation classification tasks.
- A. Architecture and Training Setup: The pretrained CED-base encoder is frozen, while a lightweight MLP projection head maps embeddings to a 256-dimensional retrieval space.The encoder produces a 768-dimensional output and is shared across query and reference branches.
- A. Architecture and Training Setup: Audio is resampled to 16 kHz and processed with the pretrained CED feature extractor.The final encoder hidden representation is average-pooled over the sequence dimension to obtain a 768-dimensional embedding.
- A. Architecture and Training Setup: The training objective is supervised contrastive loss applied to known reference–imitation pairs from the VimSketch dataset.The objective is designed to structure the embedding space for retrieval.
- A. Architecture and Training Setup: In-batch negatives are used to structure the embedding space.They are incorporated into the supervised contrastive training objective.
- A. Architecture and Training Setup: Online data augmentation is applied symmetrically to query and reference audio inputs.Each training example receives one randomly selected time-domain transformation from a curated pool discussed in Section II.
B. Implementation
The system is trained with AdamW and a cosine learning-rate schedule, using a 256-dimensional projection of the encoder output for cosine-similarity retrieval.
- B. Implementation: Training uses AdamW with a cosine schedule starting at 1 × 10−3, weight decay of 1 × 10−5, batch size 128, and 50 epochs.Evaluation is performed after every epoch, and the contrastive-loss temperature τ is fixed at 0.07.
- B. Implementation: A linear projection maps the encoder’s 768-dimensional output to a 256-dimensional embedding for cosine-similarity retrieval.
IV. SUBMISSION #2 · A. Architecture and Training Setup
Submission #2 uses a shared, AudioSet-pretrained MobileNetV3 encoder with L2-normalized embeddings for cosine-similarity retrieval. Training combines supervised contrastive learning on VimSketch positives with triplet regularization from excluded VocalSketch recordings, dynamically weighted by active semi-hard triplets.
- A. Architecture and Training Setup: The architecture uses a MobileNetV3 encoder pretrained on AudioSet.It builds on the QbVI framework proposed by Greif et al.
- A. Architecture and Training Setup: A single shared encoder processes both reference and imitation audio.This differs from the dual-tower architecture of Greif et al. and reduces parameter overhead while promoting embedding alignment.
- A. Architecture and Training Setup: L2-normalized encoder outputs enable cosine similarity for loss computation and retrieval.The normalization supports a common similarity metric across training and inference.
- A. Architecture and Training Setup: Training combines supervised contrastive learning with triplet-loss regularization.Contrastive positives come from VimSketch, while triplet negatives come from the excluded VocalSketch subset.
- A. Architecture and Training Setup: Excluded VocalSketch examples comprise practice recordings and human-rejected imitations outside standard contrastive supervision.These examples are used as negative samples for triplet construction.
- A. Architecture and Training Setup: The triplet-loss weight is dynamically scaled according to the number of active semi-hard triplets in each batch.When few valid triplets are available, the triplet contribution is down-weighted to avoid destabilizing training.
B. Implementation
The implementation follows AudioSet-style preprocessing, applies independent online augmentation to query and reference inputs, and trains with specified optimization and loss hyperparameters.
- Input preprocessing: Audio is resampled to 32 kHz and converted into 128-band log-Mel spectrograms using 10-second inputs, 800-sample windows, 320-sample hops, and 1024-point FFTs.These settings closely follow AudioSet’s preprocessing pipeline.
- Input preprocessing: Independent online augmentation is applied to query and reference inputs during training to improve robustness to vocal variation and noise.
- Input preprocessing: The penultimate layer outputs 960-dimensional embeddings.
- Optimization: Training uses AdamW with cosine learning-rate scheduling, one warmup epoch, a 2×10−4 maximum rate, a 5×10−5 minimum rate, 30 epochs, and batch size 64.
- Optimization: The contrastive temperature is fixed at τ = 0.07 and the triplet margin at 0.6.
V. RESULTS
The results use the official QVIM 2025 validation protocol on the qvim-dev set, evaluating retrieval with pair-wise and class-wise measures. Submission #2 is identified as the winning AES AIMLA 2025 Challenge submission.
- Evaluation protocol: The qvim-dev validation set contains query-reference pairings and class metadata, supporting pair-wise and class-wise evaluation.The evaluation follows the official QVIM 2025 validation protocol.
- Evaluation metrics: Mean Reciprocal Rank (MRR) over known query-reference pairs is the primary metric, with NDCG and class-wise MRR as secondary metrics.NDCG assesses ranking quality, while class-wise MRR assesses class-level retrieval consistency.
- Validation results: Submission #2 is the winning submission in the AES AIMLA 2025 Challenge on querying sound effects by vocal imitation.Table I summarizes validation performance for both submissions in comparison with the baselines.
VI. CONCLUSION
The conclusion presents two complementary approaches for Query by Vocal Imitation retrieval: frozen-encoder supervised contrastive learning and jointly fine-tuned contrastive-triplet learning. Together, the submissions emphasize transfer learning and curriculum design as effective strategies for improving retrieval over baseline systems.
- Task and approaches: The QbVI task retrieves reference-database audio samples from user-provided vocal imitations.The work was conducted as part of the AES-AIMLA 2025 Challenge.
- Submission #1: Submission #1 uses a frozen, AudioSet-pretrained CED-base encoder with supervised contrastive loss and in-batch negatives.The lightweight approach achieves strong retrieval performance with minimal fine-tuning.
- Submission #2: Submission #2 fine-tunes a shared MobileNetV3 encoder using a hybrid contrastive and triplet objective.The supplied passage identifies this as an extension of the training pipeline and introduces semi-hard negatives before truncation.
- Overall findings: Both strategies highlight transfer learning and curriculum design while yielding significant retrieval gains over baseline systems.The conclusion attributes these strategies to careful pretraining and structured negative sampling, respectively.