Expressive Guitar Synthesis via Controllable Latent Flow Matching

1. INTRODUCTION

• MIDI-only systems leave per-string pitch and amplitude trajectories unspecified, although these within-note decisions carry much of guitar performance’s expressiveness.Because the model learns the realization internally, users cannot adjust it.Source: §1 · p. 1
• Technique-label conditioning can require expert knowledge and scarce annotations, while predicted-control approaches still leave room to improve audio realism.Prior work includes VIOLET, ViolinDiff, MIDI-DDSP, and DDSP-Guitar.Source: §1 · p. 1
• The proposed two-stage system predicts per-string f0 and amplitude trajectories, then renders them with latent flow matching while using gated cross-string attention in both stages.A pseudo-velocity predictor allows inference from timed note events alone, and users can edit predicted trajectories before rendering.Source: §1 · p. 1
• The paper reports improved control fidelity over both baselines, with cross-string attention benefiting polyphonic regions and pseudo-velocity enabling MIDI-only synthesis.The approach is presented as retaining editable controls while improving synthesis realism.Source: §1 · p. 1

2. PROPOSED METHOD

• A performance predictor maps frame-wise MIDI and note velocity to per-string f0 and amplitude-envelope trajectories, which condition a latent flow-matching audio synthesizer.The synthesizer generates per-string audio and mixes the filtered signals.Source: §2 · p. 2

2.1. Gated Cross-String Performance Predictor

• An axial transformer models trajectories over time and interactions across strings, using zero-initialized gates to introduce cross-string information as it proves useful.The predictor combines pitch, voiced state, relative note position, onset, and velocity inputs; learned string-identity embeddings localize tokens by string.Source: §2.1 · p. 2, Figure 1 · p. 2

2.2. Gated Cross-String Synthesizer

• The synthesizer encodes predicted controls and MIDI context to condition an axial DiT, which predicts a vector field for generating per-string audio latents.MIDI voiced-state and onset embeddings help reinterpret regions affected by imperfect control predictions; string-specific impulse responses are applied before mixing.Source: §2.2 · p. 2, Figure 2 · p. 2

2.3. Pseudo Velocity Predictor

• A transformer predicts each note’s pseudo-velocity from MIDI features, enabling inference without ground-truth velocity input.Its note features are pitch, log-scaled duration, and preceding and succeeding inter-onset intervals.Source: §2.3 · pp. 2–3

2.4. Implementation Details

• The system extracts f0 and log-RMS envelopes on the Stable Audio Open latent time grid and trains separate predictor, synthesizer, velocity, and mixing-filter components.The performance predictor uses frame-wise, trajectory-difference, and note-wise L1 losses; the synthesizer uses rectified-flow velocity MSE and classifier-free guidance.Source: §2.4 · p. 3

3.1. Experimental Setup

• Experiments use GuitarSet’s recording-level split, compare Single-stage and DDSP-Guitar, and include an oracle-synthesizer reference conditioned on ground-truth controls.All six string recordings from a performance stay in the same split.Source: §3.1 · p. 3

3.2. Evaluation

• Evaluation measures control fidelity and spectral audio quality, and a 32-participant listening test rates content correctness, expressiveness, audio quality, and overall preference.The listening test uses 18 eight-second excerpts from the test split.Source: §3.2 · p. 3

• 26.52 cents (vs. 33.53) f0 MAE and 0.88 (vs. 0.63) envelope correlation favor the oracle-velocity variant over Single-stage.Source: §4 · pp. 3–4, Table 1 · p. 4
• Oracle-velocity synthesis leads the synthesized systems on CC (3.46), EX (3.24), AQ (3.10), and OP (3.04).Source: Table 1 · p. 4, §4 · p. 4
• Removing decoder and then all synthesizer cross-string attention progressively degrades MSSL, most in multi-string windows, under oracle controls.Source: Table 2 · p. 4, §4 · p. 4
• Predicted velocity leaves f0 Med. nearly unchanged (10.45 vs. 10.46 cents) and mainly affects the envelope, while retaining higher control fidelity than both baselines.Source: §4 · p. 4
• DDSP-Guitar obtains lower MSSL (8.27 vs. 8.79), but the proposed system scores higher on AQ and OP despite that spectral-distance result.The authors attribute the MSSL difference to DDSP-Guitar being optimized on mixed recordings while their synthesizer is supervised on isolated strings.Source: §4 · p. 4
• Single-stage beats DDSP-Guitar on control metrics but earns lower CC (2.85 vs. 3.09), suggesting frame-level metrics can miss perceptually salient errors.Source: §4 · p. 4

• The two-stage pipeline improves per-string control fidelity and synthesis quality over a single-stage system while providing editable pitch and envelope controls.Source: §5 · p. 4
• A pseudo-velocity predictor supports synthesis without note-level velocity, and cross-string attention is particularly beneficial in polyphonic regions.Source: §5 · p. 4
• Future work includes timbre and recording-environment conditioning, technique steering through intermediate controls, and string separation to reduce reliance on hexaphonic recordings.Source: §5 · p. 4