PaperlayerView exact revision on arXiv

Source-linked AI summary

MPJudge: Towards Perceptual Assessment of Music-Induced Paintings

Shiqi Jiang, Tianyi Liang, Huayuan Ye, Changbo Wang, Chenhui Li

arXiv:2511.07137v2cs.CV

TL;DR

Assessing whether paintings perceptually reflect their source music is difficult because existing approaches rely on emotion proxies and simple late-stage fusion. The paper builds an expert-annotated dataset and introduces MPJudge, which fuses music into a visual encoder and uses preference learning for ambiguous cases. MPJudge outperforms reported baselines across datasets and metrics, with qualitative analyses also supporting alignment with human judgments.

  • Problem

    Music-induced painting assessment lacks direct perceptual-coherence evaluation, while existing dual-encoder approaches use late fusion and available datasets rely on automated emotion labels.

    Source: Introduction · p. 1

  • Method

    The paper constructs MPD with expert scalar ratings and pairwise preferences, then trains MPJudge to modulate visual features with music and learn from ambiguous-case preferences.

    Source: Introduction · p. 2, Dataset · p. 3, Training Objective · p. 5

  • Results

    MPJudge outperforms state-of-the-art baselines across multiple datasets and evaluation metrics, and qualitative analysis and user studies report alignment with human perceptual judgments.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    The reported results support direct human-judged perceptual coherence as an assessment target for music-induced paintings, with MPJudge aligning with human judgments.

    Source: Introduction · p. 2, Conclusion · p. 7

    MPD uses independently sourced paintings and music because paintings created directly under musical influence are scarce and limited in quality.

    Source: Dataset · p. 3

Abstract

from arXiv · show

Music induced painting is a unique artistic practice, where visual artworks are created under the influence of music. Evaluating whether a painting faithfully reflects the music that inspired it poses a challenging perceptual assessment task. Existing methods primarily rely on emotion recognition models to assess the similarity between music and painting, but such models introduce considerable noise and overlook broader perceptual cues beyond emotion. To address these limitations, we propose a novel framework for music induced painting assessment that directly models perceptual coherence between music and visual art. We introduce MPD, the first large scale dataset of music painting pairs annotated by domain experts based on perceptual coherence. To better handle ambiguous cases, we further collect pairwise preference annotations. Building on this dataset, we present MPJudge, a model that integrates music features into a visual encoder via a modulation based fusion mechanism. To effectively learn from ambiguous cases, we adopt Direct Preference Optimization for training. Extensive experiments demonstrate that our method outperforms existing approaches. Qualitative results further show that our model more accurately identifies music relevant regions in paintings.

Introduction

Music-induced painting assessment asks whether visual art perceptually reflects its source music, but existing computational approaches rely on emotion proxies and limited cross-modal interaction. The paper introduces human-annotated data and MPJudge to assess perceptual coherence directly.

Introduction

  • The paper presents music-induced painting as an art practice translating musical rhythm, emotion, and structure into visual forms, while computational assessment remains underdeveloped.The practice is framed as a cross-modal interpretation related to synesthesia.Source: Introduction · p. 1
  • Existing emotion-based matching relies on imprecise emotion recognition and misses perceptual cues such as rhythm, timbre, spatial composition, and visual texture.Emotion models are used to estimate whether music and paintings evoke similar affective states.Source: Introduction · p. 1
  • Perceptual coherence spans low-level cues and high-level composition, semantics, and emotional tone, which simple late-stage dual-encoder interactions cannot capture.Existing datasets also rely on automated emotion annotations rather than direct human perceptual judgments.Source: Introduction · pp. 1–2
  • The framework dataset contains approximately 6,000 music pieces and 11,000 paintings, forming over 50,000 music–painting pairs, each scored for perceived coherence; domain experts additionally provided pairwise preferences for ambiguous cases near 0.5.Source: Introduction · p. 2
  • Experiments and user studies report that MPJudge surpasses state-of-the-art approaches and offers more interpretable visualizations.The contribution summary describes residual activation map visualization as the interpretability evidence.Source: Introduction · p. 2

Related Work

Prior painting-assessment work covers emotion, aesthetics, and detailed artwork attributes, while music–image matching commonly bridges modalities through emotion or content proxies. This paper instead targets direct human-judged perceptual alignment between music and painting.

Painting Assessment

  • Existing painting datasets assess emotions, aesthetics, or attributes such as expressiveness, symbolism, and creativity, but do not address whether a painting aligns with music.The cited examples include MART, JenAesthetics, VAPS, BAID, AACP, and PPJudge.Source: Painting Assessment · p. 2

Bridging Music and Painting

  • Music–image studies use emotional tags or content descriptions as intermediate representations, whereas this work connects the modalities directly through human perceptual judgments.The section describes emotional representations including dimensional and categorical emotion models, alongside content-tag and text-label approaches.Source: Bridging Music and Painting · p. 2

Music Painting Matching Models

  • Prior music–image matching uses emotion alignment or content recognition, while MPJudge integrates music features into the painting encoder through modulation-based fusion.The cited matching approaches include ACP-Net and CDCML for emotion, and a content-recognition approach; the dataset distribution is shown in Figure 2.Source: Music Painting Matching Models · pp. 2–3, Figure 2 · p. 3

Dataset

MPD pairs independently sourced paintings and music clips for expert judgments of perceptual coherence, with scalar scores and added preference labels for ambiguous cases. The section describes the data, annotation protocol, reliability analyses, and preference-pair construction.

Dataset

  • MPD represents each painting–music pair with a perceptual coherence score, using independently sourced content because directly music-created paintings are scarce and limited in quality.Paintings and music are randomly paired, and domain experts provide scalar ratings.Source: Dataset · p. 3

Data Collection

  • The painting collection combines 9,885 WikiArt works across five styles with approximately 1,200 children’s paintings, while the music collection yields 6,127 clips from about 1,000 tracks.The music tracks are segmented into non-overlapping 15-second clips.Source: Data Collection · p. 3

Data Annotation

  • Each pair receives five independent ratings; the highest and lowest scores are discarded and the remaining three averaged, producing annotations for 50,000 pairs.Annotators are domain experts, and the briefing defines consistency as alignment between the perceptual experiences evoked by music and painting, beyond simple emotional matching.Source: Data Annotation · p. 3

Annotation Analysis

  • The five raw ratings have an average standard deviation of 0.078, with 84.8% below 0.09 and 99.0% below 0.11; Krippendorff’s Alpha is 0.86.The reported dispersion and agreement analyses indicate generally consistent ratings and high inter-rater consistency.Source: Annotation Analysis · pp. 3–4

Preference Annotation for Ambiguous Pairs

  • Because many scalar scores cluster near 0.5, preference annotations target pairs with mean consistency scores from 0.4 to 0.6.The collection starts with 10,428 preference tasks and retains 5,582 music-to-painting and 5,403 painting-to-music samples after majority-consensus filtering; each task has at least three annotators.Source: Preference Annotation for Ambiguous… · p. 4

Method

MPJudge predicts perceptual coherence by conditioning a painting encoder on music features. It combines scalar-score regression with preference learning for ambiguous pairs.

Method

  • MPJudge encodes music and painting with separate encoders, fuses music features into the painting stream, and predicts perceptual coherence.The painting encoder is Transformer-based, while the music encoder is a lightweight convolutional network.Source: Overview · p. 4

Music Encoder

  • A four-block convolutional encoder turns mel-spectrogram patterns into tokens and projects them to the embedding dimension used by downstream modules.Each block uses convolution, batch normalization, and SiLU activation; the lightweight design reflects the localized patterns and lower structural complexity of mel spectrograms.Source: Music Encoder · p. 4

Painting Encoder

  • Music features modulate painting representations through MAN after each self-attention block, adapting visual features to the corresponding audio context.MAN uses painting features as content and music features to produce scale and shift parameters; its purpose differs from AdaIN-style visual-domain style transfer.Source: Painting Encoder · pp. 4–5
  • Layerwise Modulation Intensity Maps measure token-representation changes after MAN, enabling comparison of music conditioning across visual-encoder layers.The maps provide insight into audio conditioning of both low-level appearance features and high-level semantic structures.Source: Painting Encoder · p. 5

Training Objective

  • Training combines regression on scalar coherence scores with DPO on pairwise preferences, applying both losses to preference-labeled data and regression alone otherwise.The preference loss uses preferred and less-preferred candidates and a fixed reference model to anchor preference direction.Source: Training Objective · pp. 5–6

Evaluation

MPJudge outperforms baselines across datasets and metrics, aligns with human judgments in user studies, and uses DPO and modality-adaptive fusion to improve music-conditioned visual assessment.

Experimental Details

  • The setup uses 469 × 128 Mel inputs and 256 × 256 × 3 paintings with a 12-block encoder; evaluation compares three baselines across IMAC, IMEMNet, and the authors’ dataset.Training uses Adam with a learning rate of 1 × 10−5 on eight NVIDIA H100 GPUs.Source: Experimental Details · p. 6, Quantitative Experiments · p. 6

Quantitative Experiments

  • On our dataset, our method consistently outperforms all baselines across all metrics and datasets, achieving an SRCC of 0.86 and an MAE of 0.04; Accuracy uses a fixed threshold.Source: Quantitative Experiments · p. 6

User Study

  • In a 20-participant study, model predictions align well with human judgments in binary matching and five-painting ranking tasks.Figure 4 presents statistical results for the user study; the ranking task compares model and user rankings using SRCC.Source: User Study · p. 6, Figure 4 · p. 6

Ablation Study

  • DPO replaces unstable scalar-only supervision near 0.5 with relative preferences and improves every evaluation metric.Without DPO, ambiguous samples clustered around 0.5 produce unstable supervision signals and suboptimal learning.Source: Ablation Study · p. 7
  • MAN achieves the best results across fusion strategies, outperforming cross-attention while being simpler and more efficient.The comparison includes painting-only encoding, feature concatenation, cross-attention, and MAN.Source: Ablation Study · p. 7

Visualization

  • Early layers emphasize acoustically resonant textures and color patterns, while deeper layers show more global, semantic modulation.Brighter regions in Figure 5 indicate stronger modulation by music; it shows maps from the first and last three Transformer blocks.Source: Visualization · p. 7, Figure 5 · p. 7

Conclusion

The paper presents MPD, a human-annotated dataset with scalar coherence scores and pairwise preferences, and MPJudge, a music-conditioned visual assessment model. Experiments report gains over state-of-the-art baselines, while future work would broaden the visual content and settings considered.

Conclusion

  • The paper constructs a large-scale human-annotated dataset with scalar coherence scores and pairwise preferences, and proposes MPJudge using modality-adaptive normalization and direct preference optimization.The pairwise preferences are intended to better leverage relative judgments in ambiguous cases.Source: Conclusion · p. 7
  • Experiments show that MPJudge outperforms state-of-the-art baselines across multiple datasets and evaluation metrics.Source: Conclusion · p. 7
  • Qualitative analysis and user studies indicate that the model aligns well with human perceptual judgments.Source: Conclusion · p. 7
  • Future work will explore sketches or abstract art and extend the model to generative or interactive settings.Source: Conclusion · p. 7
Loading 2511.07137v2…