Source-linked AI summary
Art2Song: Enhancing Visual Art Appreciation with Contextual Music Generation
Sungeun Jo, Myung Jin, Kim, Chi Yoon Jeong
TL;DR
Art appreciation can draw on historical and art-historical context that is difficult to perceive from an image alone. Art2Song assigns visible evidence to Lyrics and contextual mood to a background soundtrack; generated examples show the components can harmonize or contrast, while the prototype also revealed reliability challenges.
Problem
Historical and art-historical context can deepen art appreciation but is difficult to perceive from an image alone, and systems do not necessarily make explicit how visual and contextual sources shape interpretation.
Source: §1 · p. 1Method
Art2Song uses separate channels to turn observable image details into Lyrics and derive background-music conditions from Contextual Mood in museum artwork descriptions.
Source: §2 · p. 2Results
Generated examples paired visual-content Lyrics and a Contextual Mood soundtrack in harmony or contrast, suggesting the two aspects can be examined separately.
Source: §2 · p. 2Takeaways & Limitations
Art2Song explores an added mode of art appreciation that pairs attention to visible details with contextual emotion through music, rather than replacing text-based descriptions.
Source: §3.1 · p. 2The prototype sometimes omitted or abstracted requested visual elements, and music generation sometimes dropped or rendered Lyrics unclearly.
Source: §2 · p. 2
Abstract
from arXiv · showhide
Art2Song is a conceptual framework that expresses artworks as sound by separating Non-Visual Context, which is difficult to perceive from the image alone, from Visual Evidence. Visual Evidence, such as objects, colors, and spatial composition, is transformed into Lyrics, while the Contextual Mood derived from the historical and art-historical context in the museum's artwork description is reflected in the background soundtrack. Rather than describing artworks textually, Art2Song aims to explore the possibility of a new mode of art appreciation in which viewers experience hidden stories and emotional context through music. As future interaction directions, we plan an Emotional Layer Blending Slider interface and a structured, traceable song-generation scenario, presenting the possibility that users can explore the relationship between Visual Evidence and Contextual Mood.
1 Introduction
Art2Song explores conveying visually hidden historical and art-historical context through sound, alongside visible details. It assigns those sources distinct song roles to bring the scene and its context together.
1 Introduction
- Objects, colors, and spatial relations become Lyrics, while Contextual Mood from historical and interpretive context shapes the background soundtrack.The proposed song lets viewers experience the scene and the context behind it together.Source: §1 · pp. 1–2
- Museum and HCI research treats musical augmentation as interpretation; viewers connect paintings and music through emotion, movement, narrative, and theme, and curated pairings can feel more meaningful than automatic ones [10].Multisensory experiences can also shape artwork interpretation.Source: §1 · p. 1
- Image-to-Music systems translate visual features into music [2], and newer systems jointly condition generation on visual and contextual information [4] [7], without necessarily making each source’s influence explicit.The introduction frames this as a gap in how the resulting interpretation is made explicit.Source: §1 · p. 1
2 Proposed Concept Prototype Pipeline
The prototype uses separate visual-evidence and contextual-affective channels to generate vocal songs. Examples demonstrate that visual Lyrics and a context-reflecting soundtrack can harmonize or contrast, while exposing extraction and lyric-preservation failures.
2 Proposed Concept Prototype Pipeline
- AIC supplies artwork identifiers, subject metadata, and curated descriptions, supporting both object verification and context-based mood extraction.The API was selected as an initial source because it provides subject metadata and curated descriptions together.Source: §2 · p. 2
- The Visual Evidence Channel uses Qwen3.5 to extract observable objects, positions, spatial relations, colors, and composition, while AIC subject metadata [1] cross-checks objects and marks their provenance.The VLM adds spatial detail beyond AIC’s main-subject and object metadata; matched objects are cross-confirmed, while unmatched ones remain marked as image-only evidence.Source: §2 · p. 2
- The Contextual Affective Channel extracts mood and rationale from the artwork description, then derives genre, BPM, instrumentation, dynamics, and vocal tone as music-generation conditions.The extraction is grounded in curatorial and art-historical interpretation and artist statements where provided.Source: §2 · p. 2
- Generated examples combined visual-content Lyrics with a Contextual Mood soundtrack in harmony or contrast, suggesting viewers could examine visual impression and context-derived emotion separately.ACE-Step 1.5 [3] generates an approximately 60-second vocal song from both channels’ outputs.Source: §2 · p. 2
- Observed failures included omitted visual elements, abstract rather than observable descriptions, and Lyrics dropped or rendered unclearly during generation.Future work will evaluate failure frequency and improve the model and generation conditions to reflect intended visual information and Lyrics more consistently.Source: §2 · p. 2
3 Expected Experience and Future Interaction
Art2Song proposes an added listening experience in which Lyrics direct attention to visible details and the soundtrack conveys contextual mood. Future interaction concepts let users vary the influence of those sources and trace how evidence informs the song.
3.1 New Way of Appreciating Art
- The proposed listening experience pairs general-to-specific Lyrics about visible details with a soundtrack conveying Contextual Mood, aiming to prompt viewers to revisit overlooked details and explore historical context.In The Bedroom, the Lyrics trace room composition while the soundtrack conveys a melancholic, unstable atmosphere tied to longing for rest and inner tension.Source: §3.1 · p. 2
3.2 Emotional Layer Blending Slider
- The conceptual, not-yet-implemented slider varies contextual influence from visual-focused (α≈0), through balanced (α≈0.5), to context-guided (α≈1), while Lyrics remain limited to observable Visual Evidence.Weighted conditions regenerate the song with ACE-Step 1.5 rather than crossfading audio; continuous weighting and parameter mapping remain future work.Source: §3.2 · p. 2
3.3 Structured and Traceable Song Generation
- The system records evidence and intermediate representations behind Lyrics and soundtrack, enabling proposed process views that trace artwork information into the song or music back to its contextual grounding.Records include image-extracted objects and spatial relations cross-checked against AIC, plus the description, extracted mood, rationale, and resulting music conditions.Source: §3.3 · p. 2
4 Conclusion
The conclusion presents Art2Song as a framework for conveying artwork visuals and hidden context through separate song components. It identifies prototype challenges and outlines future reliability work, broader sources and formats, and user studies.
4 Conclusion
- Art2Song conveys artwork visual elements through Lyrics and hidden contexts through a background soundtrack, while the initial prototype revealed challenges in visual extraction, lyric preservation, and generation control.Source: §4 · p. 2
- Future work will improve pipeline reliability, test vocal songs against narration-based soundtracks, and extend the system to additional museum APIs and public metadata sources.Source: §4 · p. 3
- Planned user studies will compare Visual Evidence-focused, Contextual Mood-focused, and Combined settings on artwork understanding, engagement, and interpretive exploration.Source: §4 · p. 3