PaperlayerView exact revision on arXiv

Source-linked AI summary

Taste-aware music retrieval from audio embeddings

Matteo Spanio, Antonio Rodà

arXiv:2607.03296v1cs.SDcs.IRcs.LGcs.MM

TL;DR

Taste-from-audio prediction is largely missing from content-based multimedia retrieval, despite established sound–taste correspondences and potential for taste-based music indexing. This paper benchmarks frozen audio encoders and finds strong prediction accuracy, with taste-space retrieval outperforming a chance-level CLAP-text baseline.

  • Problem

    Taste-based semantic indexing is largely absent from content-based multimedia retrieval despite established sound–taste correspondences and potential applications beyond genre and mood.

  • Method

    The paper benchmarks ten frozen audio encoders across four HEAR families with shared multi-task regression, retrieval evaluation, and psychophysics-grounded interpretability.

  • Results

    0.134 macro RMSE is achieved for five tastes, while taste-space retrieval outperforms CLAP-text, whose correlation and AUC sit at chance.

  • Takeaways & Limitations

    A single VGGish suits low-error rating, whereas AST+VGGish is preferable for taste-based indexing because deployment depends on the task.

  • Takeaways & Limitations

    The small evaluation bounds generalisability, while Western, sweet/bitter-centric crossmodal evidence leaves cross-cultural transfer and the spicy axis under-constrained.

Abstract

from arXiv · show

Crossmodal correspondences between sound and taste are well established in psychology and neuroscience, but largely absent from content-based multimedia retrieval. We formalise taste-from-audio prediction as a content-based music information retrieval benchmark over a perceptually validated multi-source corpus, comparing ten frozen audio encoders from the four HEAR families under a shared multi-task regression head, with gated late-fusion as a configurable variant. In order to assess the effectiveness of the models, we compute absolute error and rank correlation. The strongest systems predict the five tastes within a macro RMSE of 0.134; on held-out real music their error is less than half a single rater's deviation from the consensus (RMSE 0.13 vs. 0.28), so the model tracks the group consensus more closely than an average human rater, and well below the previous state of the art baseline (0.219). On absolute error the encoders are statistically flat, with a single VGGish matching the best fusion, but gated late-fusion's advantage is confined to rank correlation (macro Pearson r 0.724 vs. 0.666). Operationalised as a content-based retrieval index, the predicted taste space ranks a 309-item pool far more faithfully than a CLAP-text baseline, which sits at chance; ridge probes and an audio-bandstop knockout read the strongest representations against documented sound-taste correspondences.

I. INTRODUCTION

The paper frames taste-from-audio prediction as a content-based music retrieval problem, motivated by established sound–taste correspondences and the need for reproducible semantic indexing. It evaluates frozen audio encoders, retrieval performance, and psychophysics-grounded interpretability on a perceptually validated multi-source corpus.

  • Motivation: Predictable taste judgments could enable semantic music indexing, queries beyond genre and mood, and recommendations such as “a similar track but sweeter”.The task also supports explainable multimedia learning by comparing model behaviour with empirical psychology findings.
  • Benchmark: Taste-from-audio prediction extends prior regression work into a broader content-based MIR benchmark using a perceptually validated multi-source corpus and ten frozen encoders from four HEAR families.The protocol compares single-encoder and gated late-fusion forms under a shared evaluation framework.
  • Benchmark: 0.134 macro RMSE is achieved by the best systems; held-out real-music error is RMSE 0.13 versus 0.28 for a single rater and 0.219 for the previous baseline.These results are presented in the context of rating replacement.
  • Retrieval: A 309-item taste-profile retrieval pool is led on every metric by the predicted taste space, while the generic CLAP-text baseline collapses to chance.The paper presents this as the first retrieval evaluation of taste-conditioned music indexing, to the authors’ knowledge.
  • Interpretability: Psychophysics-grounded interpretation combines ridge probes to nine spectral descriptors with a cross-encoder audio-bandstop knockout to assess documented sound–taste correspondences.The analyses read the strongest representations against established crossmodal findings linking acoustic properties with tastes.

II. RELATED WORKS · A. Sound-taste correspondences and sonic seasoning

The literature establishes robust sound–taste correspondences and has developed computational methods for generating, norming, discriminating, and surveying taste-linked music. Psychophysical studies associate auditory features such as pitch, consonance, timbre, and tempo with canonical tastes, while computational work operationalises these links across music databases and models.

  • A. Sound-taste correspondences and sonic seasoning: Psychophysical studies map high pitch, consonance, and bright timbre to sweetness, while low pitch, roughness, and darker timbres correspond to bitterness.Sweet and bitter show the largest effects, with replications confirming instrument-to-taste mappings.
  • A. Sound-taste correspondences and sonic seasoning: Crossmodal auditory–taste correspondences have been documented for over a decade across pitch, consonance, timbre, and tempo.The documented mappings concern the four canonical tastes.
  • A. Sound-taste correspondences and sonic seasoning: Sound also modulates perceived taste intensity.
  • A. Sound-taste correspondences and sonic seasoning: Four computational threads address sound–taste relations: compositional, stimulus norming, discriminative, and generative or dataset-level work.
  • A. Sound-taste correspondences and sonic seasoning: 100 tracks were normed for sweet, bitter, sour, and salty tastes in a music database.
  • A. Sound-taste correspondences and sonic seasoning: Five separate Audio Spectrogram Transformer regressors were fine-tuned on a curated soundtracks corpus to label the FMA dataset at scale.

B. Audio representations, explainability, and retrieval · III. METHODOLOGY

The section situates taste-aware retrieval within modern audio-representation research, which has largely excluded sensory dimensions such as taste. It adopts ridge probes and CKA for explainability and uses attribute-controlled frozen-embedding retrieval as the downstream test bed.

  • B. Audio representations, explainability, and retrieval: Modern audio representations include supervised event, self-supervised speech and music, multimodal, codec-based, and music-pretrained models.These families motivate comparing diverse frozen representations.
  • B. Audio representations, explainability, and retrieval: HEAR aggregates benchmarks across major audio-model families but excludes sensory dimensions such as taste.A similar exclusion is noted for music emotion.
  • B. Audio representations, explainability, and retrieval: Prior work predicts 19 perceived-semantic axes of functional sounds using feature extraction and ensembles.This is identified as the closest related approach to the paper.
  • III. METHODOLOGY: Ridge probes and Centered Kernel Alignment are adopted as standard probes of what embeddings linearly preserve.The methodology uses both techniques for explainability.
  • B. Audio representations, explainability, and retrieval: Frozen music embeddings support attribute-controlled search through a learned transformation.This prior setup motivates the downstream retrieval evaluation.
  • III. METHODOLOGY: The paper uses attribute-guided retrieval as its downstream test bed and reports a static retrieval-by-taste-profile baseline at corpus scale.The baseline is described as robust for the paper's corpus scale.

A. Dataset and task · B. Models architecture · C. Audio encoders and fusion

The benchmark uses a fixed-split, multi-source music–taste corpus and predicts five normalized taste intensities with frozen audio encoders plus a shared multi-task head. It compares ten encoders spanning four HEAR families, with gated late fusion evaluated across seven configurations over the four strongest encoders.

  • A. Dataset and task: The corpus combines three audio–taste sources under an unreshuffled split of 269 train, 68 validation, and 40 test clips.Sources include a perceptually validated real-music subset, MusicGen-generated songs, and a soundtracks survey corpus.
  • A. Dataset and task: The task predicts five normalized taste intensities: sweet, bitter, salty, sour, and spicy.
  • B. Models architecture: Each 15 s clip is processed by a frozen encoder whose mean-pooled embedding feeds a two-layer MLP with hidden size 256, ReLU, dropout 0.2, and a final sigmoid.The head outputs a 5-D taste vector in [0, 1]^5.
  • B. Models architecture: Training uses AdamW with learning rate 10^-3, weight decay 10^-4, batch size 32, and early stopping after up to 50 epochs using validation macro r.Metrics are averaged over five seeds: {11, 22, 33, 44, 55}.
  • C. Audio encoders and fusion: The benchmark covers ten frozen audio encoders across supervised event/scene, self-supervised speech/music, multimodal/codec-based, and music-pretrained representation families.Named examples include VGGish, PANNs, AST, HuBERT, MERT, Omar-RQ, CLAP, EnCodec, and MULE.
  • C. Audio encoders and fusion: MFCC provides the DSP floor, while frozen encoders are the default because the task is low-N and cached embeddings support reproducible broad comparison.The evaluation also includes ast-5head-ref and five fine-tuned AST regressors as references.
  • C. Audio encoders and fusion: Gated late fusion scales each concatenated per-encoder embedding slice by a learned sigmoid weight before the MLP head, while keeping encoders frozen.Seven configurations are evaluated over the four strongest encoders, with candidates selected by complementarity rather than raw rank.

D. Explainability and retrieval probes

The section probes the strongest models for interpretable acoustic content and evaluates taste-aware retrieval using predicted five-dimensional taste vectors. Retrieval queries are derived from perceptually rated food profiles, while ridge probes test whether embeddings preserve nine acoustic descriptors linked to taste percepts.

  • Explainability: Ridge probes regress embeddings onto nine acoustic descriptors, including spectral centroid, rolloff, bandwidth, flatness, ZCR, RMS, tempo, harmonic-to-noise ratio, and contrast.The probes use 5-fold cross-validation and target cues associated with documented sound–taste correspondences, such as bright spectra with sweet and low-frequency energy with bitter.
  • Retrieval: The retrieval index ranks test items by predicted 5-D taste vectors against five named taste-profile queries.The queries correspond to sweet, bitter–sweet, salty, sour, and spicy profiles, replacing mood with taste in an attribute-conditioned music-retrieval regime.
  • Retrieval: Each query is the mean perceptually rated taste vector of foods with the corresponding dominant taste in the dataset.Examples include dessert for sweet, dark chocolate for bitter–sweet, umami-savory for salty, citrus for sour, and chili-burn for spicy.

IV. EXPERIMENTS

The experiments address encoder transfer and late fusion for taste prediction, interpretability of spectral cues, and sensitivity to input frequencies. They use source-aware evaluation metrics and paired-bootstrap significance testing to account for subjective-label noise.

  • Research questions: The experiments test encoder-family transfer, whether late fusion improves macro Pearson r over the previous SOTA, and whether predictions preserve documented sound–taste spectral cues.The linked research questions also examine model reliance on specific input frequencies.
  • Metrics: Evaluation reports per-target Pearson r, macro Pearson r, macro MAE, and macro RMSE on the held-out test split, with per-source breakdowns against inter-rater ceilings.Error metrics are foregrounded because the intended application is to replace or augment a human rater.
  • Significance: Encoder rankings use paired-bootstrap tests with 2000 resamples, while source-aware reporting addresses label-noise effects in the MusicGen-generated subset.Only rankings explicitly described as ties fail to clear p < 0.05.

V. RESULTS · A. Encoder benchmark and fusion

Ten frozen audio encoders and seven gated late-fusion variants were benchmarked under a shared multi-task head, with macro RMSE and MAE primary and Pearson correlation secondary. The best systems reached macro RMSE 0.134, while fusion mainly improved rank correlation and real-music predictions outperformed the average human rater’s deviation from consensus.

  • A. Encoder benchmark and fusion: Table I benchmarks ten frozen encoders, the previous SOTA ast-5head-ref, and seven gated late-fusion variants using five-seed means.The comparison uses a shared multi-task head on the 40-item test set.
  • A. Encoder benchmark and fusion: Macro RMSE and MAE are primary because they measure rating-unit error, while macro Pearson r is secondary for retrieval rank quality.Targets lie on [0, 1], and training uses masked MSE.
  • A. Encoder benchmark and fusion: 0.667 is the highest reported aggregate macro Pearson r among the listed single encoders, achieved by MULE, ahead of VGGish 0.666 and CLAP 0.664.The passage also lists four encoders from three families as tied at the top of the aggregate, but the provided text truncates the fourth name.
  • A. Encoder benchmark and fusion: 0.219→0.143 RMSE is achieved by replacing the previous SOTA setup with a frozen encoder, per-taste MLP, masked-MSE, and sigmoid output.Collapsing five heads into one shared multi-task head changes r from 0.663 →0.658 while using a 5× smaller head.
  • A. Encoder benchmark and fusion: 0.134 is the final RMSE after gated fusion adds the remaining improvement, with the first loss-and-output swap contributing 0.076 of the 0.085 RMSE gain.The gain transfers across AST, VGGish, MULE, and CLAP rather than being a single-encoder artefact.
  • A. Encoder benchmark and fusion: ≈0.13 RMSE and ≈0.10 MAE are the best systems’ real-music errors, less than half the single-rater ceiling of 0.280 RMSE and 0.227 MAE.This supports estimating the group rating more closely than an average human rater; the generated subset has lower agreement and no comparable per-rater error estimate.
  • A. Encoder benchmark and fusion: The only system competitive on both the real-music and generated subsets at once is identified in the source breakdown.The provided passage does not identify that system by name.
  • A. Encoder benchmark and fusion: 0.658 ± 0.010 seed-spread Pearson r and 0.143 ± 0.003 RMSE on AST are dwarfed by a roughly 0.18-wide bootstrap 95% CI for the best fusion’s macro r.The seven fusions lie inside one another’s intervals, so their ordering is indicative rather than significant; only large gaps clear paired bootstrap p < 0.05.

B. Psychophysics-grounded analysis explains why the strongest models work

Psychophysics-grounded probes show that the strongest models exploit distinct spectral mechanisms rather than a single shared cue structure. CLAP decodes spectral descriptors most strongly and remains robust to band removal, while AST and VGGISH depend on narrower frequency regions.

  • Psychoacoustic probes: CLAP leads every spectral probe column with R2 ∈ [0.78, 0.93], preserving fine spectral structure despite its contrastive audio–language objective.AST follows, then MERT, EnCodec, and MULE; the strongest performance is on brightness and bandwidth axes linked to sound–taste associations.
  • Bandstop knockout: The bandstop probe removes each of eight mel-scale bands and measures the per-(band, taste) Pearson drop ∆r after re-encoding.The knockout uses a zero-phase 4th-order Butterworth bandstop with forward–backward filtering and RMS renormalisation.
  • Bandstop knockout: AST concentrates taste signals in one or two narrow bands, including sweet at 260–610 Hz and bitter in the sub-bass.Positive ∆r for salty, sour, and spicy indicates band-localised nuisance detectors rather than taste-relevant signal.
  • Bandstop knockout: VGGISH splits dependencies between sub-bass for bitter, −0.166, and 3.9–5.6 kHz upper-mids for salty and sour, whereas CLAP remains band-robust.No CLAP removal exceeds |0.064|, consistent with its status as the strongest spectral decoder in Fig. 2.

C. Retrieval by taste profile

Taste-space retrieval turns predicted five-dimensional taste vectors into a content-based index where rank fidelity is the key criterion. Across a 309-item pool, taste-aware predictors substantially outperform CLAP-text, with AST+VGGISH achieving the strongest reported rank correlation and performance varying by query.

  • Evaluation setup: The evaluation ranks 309 items by Euclidean distance between predicted five-dimensional taste vectors and five food-profile queries, using P@k, Spearman ρ, and Pearson r.The pool combines 40 test items with 269 training items as in-distribution distractors.
  • Retrieval results: CLAP-text’s ρ (0.122) and AUC (0.484) sit at chance, surfacing a taste-relevant item for only one of five queries.Taste-space predictors saturate P@k wherever defined, so rank correlation separates systems.
  • Retrieval results: 0.693 was the strongest ρ, achieved by AST+VGGISH, showing that the regression-best fusion was not the rank-best system.Precision cannot separate the predictors, making ρ the decisive metric for this retrieval use case.
  • Per-query variance: 0.474–0.820: best-fusion per-query ρ ranged from citrus to dark-chocolate, with the highest values on sweet/bitter-dominant queries.This pattern is consistent with larger documented crossmodal effects for those taste dimensions.
  • Metric interpretation: AUC is near chance by construction, whereas test-only ρ remains the load-bearing metric and is invariant when distractors are replaced by ∼5,000 OOD FMA chunks.The distractors share the test items’ predicted-taste distribution.

VI. CONCLUSION

The paper establishes taste-from-audio prediction as a content-based MIR benchmark combining frozen encoders, source-aware evaluation, retrieval, and psychophysics-grounded interpretability. Its conclusions also emphasize limited generalisability from the small corpus and Western, sweet/bitter-centric crossmodal literature.

  • Contributions: The benchmark combines ten frozen encoders across four HEAR families with source-aware reporting, 309-item retrieval, and psychophysics-grounded interpretability.It evaluates taste-from-audio prediction over a perceptually validated corpus.
  • Performance: 0.134 macro RMSE is achieved by the best systems when predicting the five tastes.The passage frames this result in terms of rating replacement.
  • Limitations: 269 training and 40 test clips, with n = 20 per source, limit generalisability and favour simple heads.The ridge probe nearly matches the MLP, and scaling the annotated corpus is identified as the main lever.
  • Limitations: The Western, sweet/bitter-centric crossmodal literature leaves cross-cultural generalisation and the under-constrained spicy/trigeminal axis open.The passage identifies these as unresolved limitations of the literature underpinning the study.
Loading 2607.03296v1…