Source-linked AI summary
Diptych: Scoped, AI-Interpreted Comparison for Reference Listening in Music Production
Chongjun Zhong, Abhinaba Roy, Archishman Ghosh, Kejun Zhang, Dorien Herremans
TL;DR
Reference listening depends on deciding which passages to compare, yet tracks may not align and comparison can strain memory. DipTych keeps scope user-defined and presents structured features with AI interpretations. Musicians found the system helpful, though expert-rated judgment quality did not rise overall.
Problem
Musicians must choose which passages to compare across tracks that may differ in structure, while holding comparisons in memory.
Source: §1 · p. 2Method
DipTych lets users set comparison scope across whole tracks or selected segments and inspect feature categories such as vocal quality, pitch, and melody.
Source: §1 · p. 4, §2.3 · p. 8Results
Participants liked the interaction and felt supported, but expert-rated judgment quality did not rise overall.
Source: §6 · p. 27Takeaways & Limitations
The conclusion argues that AI-assisted comparison should make differences more navigable, inspectable, and actionable without replacing human judgment.
Source: §8 · p. 31The study did not measure which of the 59 extracted features matter most, and some feature readings were unreliable.
Source: §7 · p. 31
Abstract
from arXiv · showhide
Reference listening is a common strategy in music production, but current comparison tools often obscure a key human judgment: deciding what should be compared. We present Diptych, an AI-assisted system that lets users define comparison scope across whole tracks or independently selected segments, while inspecting structured audio features and scope-specific AI interpretations. We evaluated Diptych in a within-participants study with 12 musicians, complemented by source-blinded ratings from four expert listeners. Participants used the system to surface additional differences, nine of ten of which received at least partial expert support, and reported good usability and greater clarity about possible next steps. These findings suggest that AI support for creative comparison should prioritize user-defined scope, inspectable evidence, and actionable guidance, while avoiding authoritative judgments that exceed what the evidence can support.
1 Introduction
Reference listening requires musicians to choose which passages to compare, a difficult judgment complicated by mismatched track structures, working-memory demands, and challenges articulating impressions. DipTych keeps that choice with the user and supports whole-track or independently selected segment comparisons alongside structured audio features and AI interpretations.
1 Introduction
- Reference listening is part of professional mixing and mastering, but choosing which passages correspond is itself part of the listening task.Tracks can differ in tempo, arrangement, length, and structure, so comparable sections may not align.Source: §1 · p. 2
- Comparing passages strains working memory and requires musicians to turn vague impressions into specific, checkable judgments.The difficulty grows as comparison points multiply, while sound vocabulary remains a challenge in professional training.Source: §1 · pp. 2–3
- Whole-track comparison tools report full-recording differences but leave engineers unable to choose which passages should be compared.AI can describe differences fluently and accurately yet still address the wrong scope or fail to help users decide what to do next.Source: §1 · p. 3
- DipTych lets users compare whole tracks or independently selected segments, inspect structured features alongside AI interpretations, and accept, revise, or reject verdicts.Selected segments need not share timestamps, durations, or structural labels; timelines help users navigate both tracks.Source: §1 · pp. 3–4
- DipTych is used to derive reusable design principles: comparisons should fit user-chosen scope, expose evidence beside interpretation, and guide actionable decisions.The paper frames these as broader questions for AI-assisted creative comparison, including video editing and writing.Source: §1 · pp. 4–5
- The study examines how musicians’ judgments change, how they use system interpretations, and whether experts support those judgments.Twelve musicians completed a single-session study, while four expert listeners assessed claims by listening without the interface.Source: §1 · p. 4
2 Related Work
Prior work supports AI-assisted music creation, reference-conditioned production, and interpretable audio analysis, but leaves comparison scope and its organization largely unresolved. DipTych keeps scope user-defined and structures evidence into production-relevant dimensions.
2.1 AI-Assisted and Co-Creative Music Tools
- AI music tools often generate content or decide what to describe, whereas DipTych leaves comparison choice and judgment with the user.This follows guidance that users should be able to inspect, correct, and dismiss AI output rather than accept it wholesale.Source: §2.1 · pp. 5–6
2.2 Comparison and Reference-Conditioned Support in Music Production
- Whole-track comparison tools report broad frequency balance, loudness, or stereo differences but cannot compare independently chosen local passages.Other reference-conditioned systems instead modify the target, while conversational mixing advice still lets the model decide what to discuss.Source: §2.2 · pp. 6–7
2.3 A Structured Vocabulary for What Gets Compared
- Eight interpretable production dimensions organize comparisons at a grain producers can inspect and connect to decisions, rather than collapsing differences into one score.The dimensions draw on practitioner vocabulary, MIR facets, and production-knowledge taxonomies; vocal quality remains distinct because generic timbre descriptions miss interpretive and affective information.Source: §2.3 · pp. 7–8
- Example features make differences checkable across dynamics, timbre, expression, vocal quality, production, melody, harmony, and rhythm.Examples include integrated LUFS, spectral centroid, micro-timing feel, formants, stereo width, vibrato rate, harmonic change, and swing ratio.Source: §2.3 · pp. 9–10
- Specialized tools address isolated domains, while captioning collapses audio into free text and MERIT supplies retrieval-oriented similarity scores rather than an inspectable scoped comparison.The paper frames its user-scoped, interpretable dimensions as extending disentanglement from a fixed embedding space to features producers can read and act on.Source: §2.3 · pp. 7, 10
3 System Design and Implementation
DipTych keeps comparison scope under user control and combines structural navigation, feature comparisons, and AI interpretation in one workspace. Users can compare whole tracks or independently selected passages, with feedback tied to the current scope and intended to inform—not replace—their judgment.
3.1 System Overview and Architecture
- DipTych makes comparison scope an explicit user choice, supporting whole tracks or passages at different times and durations.Its design goals pair scope revision and structural navigation with comparative accounts that do not substitute for listening or judge musical quality.Source: §3.1 · p. 11
3.2 Comparison Interface and Interaction
- The shared workspace combines target and reference timelines, feature tables, listening, and an AI verdict across Whole Song and Selected Segments modes.Both modes use the same feature categories, allowing movement from broad to local comparison without a different display.Source: §3.2 · pp. 11–12, §3.1 · p. 11, Figure 2 · p. 12
3.2.1 Whole-Track Comparison
- Whole Song mode compares complete recordings, but a whole-track summary can hide within-track changes such as a loud chorus in an otherwise quiet track.Users can inspect categories and listen again, or move to Selected Segments to examine relevant passages.Source: §3.2.1 · p. 12
3.2.2 User-Selected Segment Comparison
- Users select each segment independently, so compared passages may differ in start time, duration, and structural label.Structural segmentation supplies navigation landmarks; changing either selection updates the feature comparison and interpretation for the new pair.Source: §3.2.2 · pp. 12–13, Figure 3 · p. 13
3.3 AI Interpretation
- AI feedback summarizes feature categories and the overall comparison, while feature values remain available for closer inspection.The interpretation describes only the current whole-track or selected-segment scope and refreshes when selections change.Source: §3.3 · pp. 13–14
- The verdict is intended to help users decide whether to change, try, or leave something, not to rank tracks or require matching the reference.Users can weigh it against feature values and their own impressions, and accept, revise, or set it aside.Source: §3.3 · p. 14
3.4 Implementation
- The implementation extracts 59 audio features in eight categories and computes scope-specific differences from independently processed track or segment records.A language model receives those differences and feature definitions to return category summaries and an overall verdict; structural labels are navigation cues, not a trained form classifier.Source: §3.4 · pp. 14–15
4 Method
The study compared musicians’ unaided judgments with their judgments after using DipTych, then examined those changes against source-blinded expert ratings. The analysis was descriptive and treated expert listening as a perceptual reference, not absolute ground truth.
4.1 Experiment Design
- Participants recorded unaided judgments before exploring DipTych, then reviewed, revised, withdrew, or added claims; this design evaluates the complete system, not the AI alone.Participants were told that AI output could be incomplete, inaccurate, or inappropriate for their creative judgment.Source: §4.1 · pp. 17–19
- Four experts rated anonymized claims independently using a three-level rubric for audibility, direction, and temporal location where applicable.The ratings served as a perceptual reference rather than absolute ground truth; disagreements were retained rather than adjudicated.Source: §4.1 · pp. 20–21
4.2 Participants
- 12 musicians completed one predetermined target–reference comparison each, with three musicians assigned to each of four track pairs.The task did not require participants to upload music or make edits in a digital audio workstation.Source: §4.1 · p. 17, §4.2 · p. 21
4.3 Analysis Methods
- The paired analysis matched judgments on the same topic, excluded entries without explicit follow-up or with changed topics, and analyzed new judgments separately.Questionnaire responses were summarized using counts and descriptive SUS statistics; open-ended feedback contextualized reported benefits and difficulties.Source: §4.3 · p. 22
5 Results
In a descriptive study of 12 musicians, DipTych was associated with additional judgments that experts often supported, while expert support for existing judgments changed little. Participants generally found the system helpful and usable, but reported limits in interpreting and verifying its analysis.
5.1 Results Overview
- The dataset contained 89 distinct judgments and 178 expert ratings; the primary before-and-after analysis used 38 topically matched pairs.The paired analysis excluded one post-use judgment with a changed topic and initial judgments without explicit follow-up.Source: §5.1 · p. 23
5.2 Changes in Comparative Judgments and Expert Support
- 9 of 10 newly added judgments received at least partial support from both experts, while full support for existing judgments shifted from 63.2% before use to 60.5% after.For existing judgments, 32 of 38 were at least partially supported by both experts at each stage.Source: §5.2 · pp. 23–24
5.3 Responses to System Interpretations
- Participants selectively accepted and challenged system interpretations, including questioning whether cited metrics justified claims and noting missing evidence.Across six review entries, four participants explicitly expressed disagreement or identified missing evidence.Source: §5.3 · p. 24
- Among 35 judgments with complete confidence ratings, confidence increased for 15, but changes did not consistently correspond to expert support.Of the 15 judgments with increased confidence, seven retained full support from both experts, seven remained below that threshold, and one lost full support from one expert.Source: §5.3 · p. 24
5.4 Perceived Benefits and Difficulties in Using the System
- The mean SUS score was 74.79 (SD = 9.26, range = 62.5–97.5), above the cited benchmark of 68; 11 participants rated the system somewhat or very helpful.All participants agreed that playback and track switching were convenient and the information load manageable.Source: §5.4 · pp. 24–25, Figure 5 · p. 25, Figure 6 · p. 25
- All 12 participants reported greater clarity about possible next steps, while eight agreed the evidence was sufficient to verify system statements.Participants also reported difficulty interpreting technical terms and charts, and four agreed that the system sometimes made insufficiently supported claims too confidently.Source: §5.4 · pp. 26–27, Figure 8 · p. 26
6 Discussion
Participants found Diptych helpful and clearer about possible next steps, but expert-rated judgment quality did not rise overall. The discussion frames its value as helping creators organize attention and decide what merits a closer listen or a change.
6.1 Multi-scale comparison turns listening impressions into inspectable questions
- Whole Song mode provides overall orientation, while Selected Segments supports closer inspection and a return to the whole track.All 12 participants said they were clearer about what to check, try, or leave unchanged; several valued segmentation and time markers for easing memory demands.Source: §6.1 · pp. 27–28
6.2 Perceived support without a measured gain in judgment quality
- 24 of 38 matched judgments had full support from both experts before system use, compared with 23 after use.The authors distinguish sharper, better-timed judgments from judgments more likely to match what experts hear.Source: §6.2 · p. 28
- 4 of 10 new judgments had full support from both experts, while 9 of 10 had at least partial support from both.Participants also reported trouble with some terms, charts, and interpretations; only 2 of 12 felt pushed to move their track toward the reference against their judgment.Source: §6.2 · p. 28
6.3 The system described detail it could not ground
- Experts flagged fluent interpretations that described details not supported by the audio, including a vocal part in instrumental music.The authors recommend tying each verdict sentence to its supporting feature or measurement and staying quiet about undetected dimensions.Source: §6.3 · pp. 28–29
6.4 More fluent judgments were not more accurate judgments
- Later judgments were often longer and more technical, but this added detail did not bring a matching rise in expert support.The authors warn that polished language can make weakly grounded judgments harder to recognize and recommend showing evidence strength alongside wording.Source: §6.2 · p. 28, §6.4 · p. 29
6.5 Supporting evidence-informed judgment while preserving creative intent
- Users should decide which differences matter and which to keep, rather than treating every difference as a fault.Claims about measurable attributes such as tempo, loudness, and key tended to draw agreement, while feel and emotion drew more mixed responses; evaluations should weigh reliability, evidence quality, and usefulness together.Source: §6.5 · pp. 29–30
6.6 Pointing earned more trust than concluding
- Participants trusted navigation features that helped them listen more than written verdicts that interpreted the music for them.The authors distinguish usability from output reliability: a participant with the highest usability score and high trust also encountered a false vocal reading.Source: §6.6 · p. 30
7 Limitations and Future Work
The expert ratings allow individual interpretation and should be read as assessments within this evaluation framework, not final measures of correctness. The feature set also varies in reliability and usefulness, and the study did not establish which features matter most to producers.
7 Limitations and Future Work
- Expert ratings allow individual interpretation, so a judgment lacking full support from both experts is not necessarily wrong.The authors frame these ratings as assessments of support within their evaluation framework rather than final measures of correctness.Source: §7 · pp. 30–31
- The 59 extracted audio features vary in reliability and usefulness, and the study did not measure which features matter most.Features came from literature rather than formative work with producers; future work includes validating extraction against reference annotations and studying which features producers want to see.Source: §7 · p. 31
8 Conclusion
DipTych keeps the choice of comparison scope with the user while supporting interpretation across whole tracks and selected segments. In a study with 12 musicians and source-blinded ratings from four experts, participants surfaced additional largely supported differences and clarified possible next steps.
8 Conclusion
- DipTych keeps the choice of what to compare with the user across whole tracks and selected segments.The system supports users in interpreting differences at both scopes.Source: §8 · p. 31
- In a study with 12 musicians and four expert listeners, participants surfaced additional largely expert-supported differences and clarified what they might check, try, or leave unchanged.Expert support for existing judgments remained largely stable, and participants sometimes questioned or rejected the system’s interpretations.Source: §8 · p. 31
- The authors argue that creative-comparison systems should expose evidence and treat AI outputs as resources for judgment, not authoritative verdicts.Deciding what a difference means and whether it matters should remain with the user.Source: §8 · p. 31