Source-linked AI summary
Dissonance Spectrum explicitly models perceptual frequency interactions for better music understanding
Tianle Wang, Xinyi Tong, Liangke Zhao, Jishang Chen, Sirui Zhang, Haoxin Zhang, Xin Jin, Duo Xu, Xiaobing Li, Song-Chun Zhu
TL;DR
Music representations often describe spectral energy without explicitly exposing relations among simultaneous frequency components. The paper introduces Dissonance Spectrum, which applies a harmonic-distance kernel to magnitude CQT data and localizes pairwise interactions; across controlled tests and six-seed evaluations, DS shows the reported strongest comparative performance while remaining a limited perceptual model.
Problem
Existing spectral magnitudes mainly mark active components, while learned embeddings entangle attributes and do not explicitly localize simultaneous relations under a specified consonance–dissonance model.
Method
DS applies a tolerance-based rational pitch-relation kernel with logarithmic harmonic distance to magnitude CQT data, attributes weighted pair relations to time–frequency locations, and injects the resulting maps through a lightweight adapter.
Results
Across six paired seeds, DS has higher mean performance than the unchanged baseline, parameter-matched Gaussian branch, and architecture-matched magnitude-CQT branch across the reported music understanding and emotion-recognition evaluations.
Takeaways & Limitations
DS provides an interpretable, complementary representation that makes one class of simultaneous frequency relations visible to researchers and downstream encoders.
Takeaways & Limitations
DS does not establish a complete consonance or musical-preference model, omitting several perceptual and learned factors; evaluation is also limited to two host families and no new listening study.
Abstract
from arXiv · showhide
Conventional music representations describe acoustic energy over time and frequency but do not explicitly expose relations among simultaneous frequency components. We introduce the \emph{Dissonance Spectrum} (DS), a nonnegative time--frequency representation that applies a tolerance-based rational pitch-relation kernel with logarithmic harmonic distance to a constant-Q spectrum and attributes aggregate pairwise interactions back to individual frequency bins. Controlled music-theory tests show strong ordinal agreement for intervals, harmonic-function connections, and church modes, and weaker but significant agreement across diverse chord voicings. DS is then encoded by a lightweight parallel branch whose zero-initialized residual projection preserves the baseline function at initialization. Across six paired training seeds in open-ended music question answering and categorical and dimensional music emotion recognition, DS obtains the highest mean on every reported endpoint relative to the unchanged baseline, a parameter-matched Gaussian-input branch, and an architecture-matched magnitude-CQT branch. These results support DS as an interpretable, complementary representation, while listener-specific perception and broader task coverage remain open problems.
Introduction
The paper introduces Dissonance Spectrum as an interpretable representation that makes selected simultaneous frequency relations explicit in music signals. It combines a harmonic-distance kernel with localized attribution and evaluates DS as a complementary inductive bias for music understanding.
- Motivation: DS addresses the gap between energy-based spectra or entangled embeddings and explicit localization of simultaneous frequency relations.The motivation concerns relations linked to consonance, dissonance, stability, tension, and emotion.
- Method: DS applies a tolerance-based rational pitch-relation kernel with logarithmic harmonic distance to magnitude CQT data and attributes weighted pair relations to time–frequency locations.The representation is deterministic and uses frequency-axis correlation for efficient computation.
- Validation: Controlled tests recover strong ordinal agreement for intervals, functional connections, and church modes, with weaker but significant agreement across diverse chord voicings.These tests assess whether the operator follows predefined music-theory orderings.
- Evaluation: Across six paired seeds, DS achieves higher mean performance than the unchanged baseline, a parameter-matched Gaussian branch, and an architecture-matched magnitude-CQT branch.The experiments span open-ended music question answering and categorical and dimensional music emotion recognition.
- Interpretability: DS preserves bin-specific attribution, so modeled interactions remain traceable to their time–frequency locations rather than collapsing into a frame-level scalar.Target-bin factors preserve register and instrumentation cues in the resulting map.
- Limitations: DS is a perceptually motivated low-level cue rather than a complete model of consonance or musical preference.Its scope excludes auditory-filter bandwidths, masking, learned tonal syntax, and cultural or individual preference.
Experiments
Controlled tests evaluate whether DS recovers musical structure, while paired-seed experiments compare DS with baseline and matched controls across MusicQA and emotion recognition.
- Experimental design: All four downstream conditions share splits, excerpts, masks, minibatch order, optimization, early stopping, checkpoint selection, and evaluation across six paired seeds.The baseline has 4.21M trainable parameters for MU-LLaMA and 1.07M for Music2Emo; matched branches have 5.57M and add 0.21M, respectively.
- Controlled validation: The validation suite tests intervals, chord qualities, functional connections, and church modes against predefined ordinal structures.It uses rendered piano audio and rank comparisons based on Spearman ρ and Kendall τb.
- Controlled validation: Intervals show the strongest agreement: C–C♯ is maximal, while the tritone is high and the perfect fifth and octave are among the lowest.Audio DS maxima track both the sampled dissonance curve and predefined ranks.
- Controlled validation: Four chord exemplars follow major < minor < suspended < diminished, but 13 voicings show positive, nonmonotonic associations shaped by spacing and inversion.Tonic connections are generally lower than predominant and dominant connections; Ionian is near the low end and Locrian highest among modes, with local reversals.
- Downstream experiments: DS has the highest six-seed mean on every MusicQA metric, with BERTScore-R increases of .0072 over Baseline, .0074 over Gaussian, and .0028 over CQT.All six paired seed differences are positive, but the authors emphasize direction and effect size over familywise distribution-free significance.
- Downstream experiments: DS has the highest mean on every Music2Emo endpoint, with R2VA increases of .0116 over Baseline, .0122 over Gaussian, and .0047 over CQT.Average valence R2 rises from .5702 to .5861, and arousal R2 from .7244 to .7317.
Conclusion
DS makes modeled simultaneous-frequency relations explicit while preserving time–frequency attribution and host-system interfaces. Results support it as an interpretable complement, but its perceptual and task scope remains limited.
- Conclusion: DS reorganizes magnitude-CQT information so modeled pair relations become visible to researchers and downstream encoders without adding a new observation.A tolerance-based harmonic-distance kernel localizes amplitude-weighted relations and uses frequency-axis correlation instead of a quadratic pair tensor.
- Conclusion: A shape-preserving, zero-initialized gated residual adapter injects DS into existing systems while preserving host outputs, heads, and baseline function at initialization.The retained time–frequency layout also supports inspection of when and where modeled relations are concentrated.
- Conclusion: DS provides a diagnostic basis for theory-conditioned probing and model comparison alongside existing audio encoders.Its practical value outside the evaluated tasks remains to be tested.
- Limitations: The evidence does not establish DS as a complete model of consonance or musical preference.The kernel omits several auditory, tonal, rhythmic, cultural, and individual-preference factors; no new listening study was conducted, and evaluation covered two host families.
Supplementary Material: Beyond Frequency: Dissonance Spectrum for
The paper is titled “Perceptually Motivated Music Understanding” and is authored by Tianle Wang and collaborators from three Chinese institutions.
- The work is titled “Perceptually Motivated Music Understanding.”
- The listed authors are Tianle Wang, Xinyi Tong, Liangke Zhao, Jishang CHEN, Sirui Zhang, Haoxin Zhang, Xin Jin, Duo XU, Xiaobing Li, and Song-Chun Zhu.
- Affiliations include the Beijing Institute for General Artificial Intelligence, Central Conservatory of Music, and Peking University.
Reproducibility Package
The separately submitted Code and Data Archive packages the implementation, data metadata, training and evaluation artifacts, and regeneration scripts for the reported results.
- The archive contains the DS implementation, extraction configuration, cached-feature schema, dataset manifests, audio hashes, training configurations, environment locks, and checkpoint rules.
- It includes evaluation scripts, seed-level predictions, and scripts that regenerate every reported table.
- Instructions cover obtaining public datasets and pretrained models from their cited official sources.
Full Derivation and Extensions of Dissonance
The derivation constructs intrinsic and cross-Dissonance Spectra by sliding a pitch-difference relation vector over CQT magnitudes. Alternative references, temporal averaging, tonic–intrinsic combinations, and correlation-based implementations extend the same attribution framework.
- Intrinsic DS: The complete representation applies a pitch-difference relation vector to CQT frames and returns target-aligned frequency locations.The implementation preserves the original frequency order while computing each frame independently.
- Efficient computation: Frequency-axis correlation replaces explicit pairwise attribution while retaining K × T outputs and avoiding O(K^2T) intermediate storage.This enables batched computation with standard correlation operators.
- Cross-DS: Cross-DS computes relations between a target spectrum and an arbitrary reference, with intrinsic DS as the special case where target and reference coincide.Fixed references can encode tonic templates or instrument tones.
- Extensions: Both intrinsic and cross-DS retain K × T shapes under the frequency-axis convention, while optional temporal DS averages references from preceding frames.A tonic–intrinsic combination is also defined for controlled analyses.
- Experimental variants: The downstream comparisons use intrinsic DS, whereas controlled tests select tonic, chord, or tonic–intrinsic references to isolate different relations.Reference choice is therefore part of the experimental convention rather than a single fixed representation.
Controlled Music-Theory Validation Details
Controlled rendered-piano tests evaluate whether DS recovers predefined ordinal structures for intervals, chord qualities and voicings, functional connections, and church modes. Agreement is strongest for modes and connections, while chord voicings show positive but locally reversed associations.
- Chord qualities: The four representative chord classes are perfectly ordered as major < minor < suspended < diminished.With n = 4, this is treated as an ordering check rather than a meaningful significance test.
- Chord voicings: Spearman ρ = .62637 (p = .02199) and Kendall τb = .46154 (p = .03048) across 13 voicings show positive chord-quality association with local reversals.DS responds to realized spacing and inversion rather than assigning one constant to each chord label.
- Functional connections: Spearman ρ = .79373 (p = .03310) and Kendall τb = .65465 (p = .05363) recover the overall tonic-function connection trend.The substantial difference between vii◦ and V7 shows that group membership does not determine complete spectral value.
- Church modes: Spearman ρ = .89286 (p = .00681) and Kendall τb = .80952 (p = .0107) correlate church-mode total DS with the predefined ordinal order.The trend is strong but not strictly monotonic because Dorian lies below Lydian.
- Exploratory analyses: The seven-instrument and 24-TET analyses are descriptive comparisons, with DS values sensitive to partial amplitudes, envelopes, noise, and continuous 50-cent steps.No universal timbre or experiential ranking is imposed.
- Level sensitivity: β = −.000 and Delastic = 1.000 after rounding, while DS peak variation is .94% relative standard deviation and 2.49% relative range across seven level conditions.Under normalized contexts, the check supports near gain-invariance over the tested range rather than universal independence from perceptual loudness.
Detailed Configuration
Downstream experiments share preprocessing, temporal alignment, encoders, fusion location, optimization, and parameter counts across Gaussian, CQT, and DS branches. Separate parameter budgets and fixed data-split conventions are specified for MU-LLaMA and Music2Emo.
- Shared preprocessing: All downstream branches use shared 24-kHz, 45-second excerpts, a 576-bin magnitude CQT grid, excerpt-level global-maximum normalization, and log(1 + D) DS compression.CQT and DS also share pooling, masks, encoders, fusion location, optimizer, schedule, and checkpoint selection.
- MU-LLaMA: MU-LLaMA has 4,205,568 baseline parameters, while Gaussian, CQT, and DS branches each total 5,565,761 trainable parameters.The matched branch adds 1,360,193 parameters through convolution, temporal encoding, attention, and gated residual fusion.
- Music2Emo: Music2Emo has 1,071,617 baseline parameters, while each Gaussian, CQT, and DS condition has 1,280,962 trainable parameters.The fixed 70/15/15 track-level split is generated once and retained across conditions.
Evaluation Implementation
The evaluation implementation specifies tokenization, smoothing, matching, metric variants, and averaging conventions for MusicQA and emotion-recognition tasks. These details distinguish the archive’s evaluator from released scoring code and define how reported metrics are computed.
- MusicQA: MusicQA uses NLTK word-punctuation tokenization before sentence BLEU with uniform weights and method-1 smoothing.The evaluator differs from the released MU-LLaMA scoring script.
- Text metrics: METEOR uses exact, stem, and WordNet synonym matching, while ROUGE-L reports stemming-based F-measure and BERTScore uses roberta-large recall.Perplexity is exponentiated per seed before averaging.
- Music metrics: MTG-Jamendo PR-AUC and ROC-AUC are macro-averaged over 56 tags.The Music2Emo weighted binary cross-entropy uses class-dependent positive and negative weights.
Additional MusicQA Comparisons
Additional MusicQA comparisons use corpus-level, six-seed evaluations rather than selected fluent examples. DS is compared with matched controls, while audit examples provide qualitative context without replacing aggregate evaluation.
- Table 7 reports DS-minus-control differences using six-seed means on a fixed MusicQA evaluation set.Negative values indicate improvements for loss and perplexity.
- The Gaussian comparison indicates that DS improvement is not explained by the added parameter budget alone.
- The architecture-matched CQT comparison suggests the pitch-resolved input accounts for part, but not all, of the gain.
- Automatically generated references may reward paraphrase overlap and do not establish factual correctness or expert-level harmonic reasoning.The paired BERTScore-R endpoint is treated as the prespecified aggregate comparison.
- Qualitative audit outputs tend to replace generic or vague descriptions with more specific acoustic, stylistic, and affective attributes.These examples are audit material and do not replace aggregate paired evaluation.
Exploratory Mechanism Analysis
Exploratory mechanism analyses perturb temporal organization, pooling, insertion point, and frequency correspondence to probe the DS configuration. All perturbations reduce the mean, but the results support consistency with these factors rather than isolating a single causal mechanism.
- Global pooling removes temporal localization, while temporal shuffling destroys order without changing marginal token statistics.
- Pre-projection and early fusion alter the insertion point, while kernel randomization preserves symmetry and value distribution but permutes frequency correspondence.
- All perturbations reduce the mean relative to the proposed ordered temporal configuration.
- These findings are consistent with contributions from frequency correspondence and temporal organization but do not isolate a single causal mechanism.
- The rows are not treated as a direct reproduction comparison because both the evaluation set and metric implementation differ.
- Holm adjustment uses one three-comparison family per task for the paired tests.