PaperlayerView exact revision on arXiv

Source-linked AI summary

Silent Metronome: Rhythmic Grounding for Live Music Accompaniment

Kevin Bretz, Derya Soydaner, Aske Plaat

arXiv:2609.07688v1cs.SD

TL;DR

Strictly causal accompaniment must generate each frame before hearing the input it accompanies, and its reliance on its own imperfect past degrades rhythmic alignment. Silent Metronome supplies an external metrical signal and auxiliary objectives; with the dataset grid, beat alignment improves over the causal baseline and surpasses a reference with a full second of look-ahead.

  • Problem

    Strictly causal accompaniment commits each output frame before hearing its input and relies on its imperfect past, while the paper asks whether an external metrical grid can restore alignment without look-ahead.

    Source: §I · p. 1

  • Method

    Silent Metronome supplies per-frame beat and bar phase with tempo and meter as an external conditioning channel, alongside auxiliary heads including future-token prediction.

    Source: §I · p. 1, B. The Silent Metronome · p. 3, C. Latent Shaping and Target-Token… · p. 3

  • Results

    Beat-F reaches 0.432 with both extensions, a factor of 3.2 over the causal baseline's 0.133 and above the fv=+50 non-causal reference's 0.269.

    Source: C. Results and Ablations · p. 4

  • Takeaways & Limitations

    Because the metrical signal is independent of generated audio, it cannot drift, suggesting rhythm can be supplied to streaming accompaniment rather than inferred from audio.

    Source: §V · p. 4

    The experiments use a grid aligned with MIDI-generated data, leaving the model's response to a live partner drifting from the set rhythm untested.

    Source: D. Discussion and Future Work · p. 4

Abstract

from arXiv · show

Live accompaniment models generate music for an incoming audio stream, committing to each output frame before hearing what comes next. In this strictly causal setting the model must infer tempo, meter, and metrical phase from its own imperfect past, whereby compounding errors quickly become audible as rhythmic drift. Put simply, the model has ears but no temporal reference, so when the ears hear imperfect, ambiguous music, the model will produce a flawed output. We propose Silent Metronome (SiMe), which gives it the temporal reference, encoding the phase within the beat and within the bar as periodic functions, pairing them with tempo and time signature, and supplying the result as a separate conditioning channel. Because this reference is independent of the generated audio, it cannot drift. Complementary auxiliary heads shape the latent representation, including a novel head that predicts the model's own future tokens. With the metrical signal taken from ground-truth annotations, beat alignment improves by a factor of 3.2 over the strictly causal baseline and surpasses a non-causal reference granted a full second of look-ahead. Coherence between input and accompaniment stays within a single point of that reference. These results suggest that streaming accompaniment systems should treat rhythm as a signal to be shared, as human ensembles do, rather than inferred.

I. INTRODUCTION

The paper targets rhythmic drift in strictly causal live accompaniment, where each frame is generated before its corresponding input is heard. Silent Metronome supplies an external metrical signal and adds auxiliary objectives to improve alignment and coherence.

I. INTRODUCTION

  • Interactive music tools can give artists expressive freedom by complementing their skills as collaborative partners.The introduction contrasts these systems with prompt-based generation, which compresses the creative process into language.Source: §I · p. 1
  • In strictly causal accompaniment, committing frames before hearing their input makes the model rely on its imperfect past, degrading rhythmic alignment.The paper asks whether an external metrical grid can restore alignment without look-ahead.Source: §I · p. 1
  • Silent Metronome adds per-frame metrical-phase conditioning and a future-token auxiliary head; with the dataset grid, the combined extensions more than triple beat alignment over the causal baseline.The framework's architecture, codec, and training recipe remain unchanged, isolating the added signal and objectives.Source: §I · p. 1

II. RELATED WORK

Prior live accompaniment systems address online generation, but none provide an explicit periodic reference for the model's position in the bar. Silent Metronome adapts cyclic metrical conditioning to the strictly causal streaming setting.

II. RELATED WORK

  • Earlier online chord systems use symbolic data on a fixed grid, while streaming audio accompaniment has been developed in systems including Wu et al. [9], LiveBand [10], and Karchkhadze and Dubnov [11].Reported scores across these systems are not directly comparable because codecs, metric implementations, and evaluation windows differ.Source: §II · pp. 1–2
  • Prior streaming work finds that look-ahead benefits beat alignment and input-output coherence, while removing it makes generation rely more on its own history.The cited systems do not give the model an explicit periodic reference for its position in the bar.Source: §II · pp. 1–2
  • Autoregressive drift accumulates errors as generation conditions on its imperfect past; existing remedies include training-time noise and scheduled-sampling approaches, –[15].LiveBand [10] instead removes feedback and generates from noise and the input mix.Source: §II · p. 2
  • Periodic conditioning is used in offline generation and other phase-locked systems, while this work adapts it as a per-frame anchor against drift in causal streaming.The method injects conditioning through adaptive normalization, modulated per frame rather than once per sample.Source: §II · p. 2

A. Background

The model uses a prefix-decoder streaming framework and receives a per-frame metrical representation that encodes beat and bar position alongside tempo and meter. The signal modulates decoder blocks and can be precomputed without changing attention or KV-cache costs.

A. Background

  • The streaming framework autoregressively generates accompaniment tokens from live audio, with future visibility controlling simulated latency or look-ahead and chunk size fixed at 50 frames.The experiments prioritize strictly causal operation at fv = 0 and use fv = +50 as a non-causal reference.Source: A. Background · p. 2

B. The Silent Metronome

  • Beat and bar phases encode each frame's position within a beat and bar, using sine-cosine pairs for continuous, unambiguous positions across boundaries.The bar phase completes one cycle per bar, and beat-phase positive-gradient zero-crossings mark beat boundaries.Source: B. The Silent Metronome · pp. 2–3, Figure 2 · p. 3
  • The 24-channel conditioning vector adds local and window-level tempo, tempo and meter change flags, and learned time-signature embeddings to the phase channels.The dataset's symbolic representation supplies the metrical grid during training and evaluation.Source: B. The Silent Metronome · p. 3
  • AdaLN-Zero injects the metrical signal into every decoder block, allowing per-frame modulation that is independent of generated audio and thus immune to autoregressive drift.The 278M-parameter increase adds roughly 5% to per-chunk generation time; phase-stripped conditioning remains at baseline alignment despite identical parameter count.Source: B. The Silent Metronome · p. 3

C. Latent Shaping and Target-Token Future Auxiliary Heads

Auxiliary heads shape the decoder's hidden representation by predicting target-stem content and future output tokens. The future-token objective asks the representation to be predictive beyond the immediate next token.

C. Latent Shaping and Target-Token Future Auxiliary Heads

  • Two jointly trained MLP heads predict the target stem's active pitches and 84-bin constant-Q spectrum, providing auxiliary objectives for musical structure.These heads are dropped at inference.Source: C. Latent Shaping and Target-Token… · p. 3
  • A third head predicts output DAC tokens 10, 25, and 40 frames ahead, corresponding to 0.2, 0.5, and 0.8 seconds, to shape predictive representations.The prediction loss averages per-codebook cross-entropy over valid positions.Source: C. Latent Shaping and Target-Token… · p. 3
  • The total training objective combines the token and auxiliary losses with all weights set to 1.The auxiliary heads are jointly trained and omitted at inference.Source: C. Latent Shaping and Target-Token… · p. 3

A. Dataset

The study uses Slakh2100, a MIDI-synthesized multitrack dataset whose symbolic source provides exact metrical grids. Each example uses one held-out stem as the accompaniment target and the remaining stems as input.

A. Dataset

  • Slakh2100 contains 2,100 multitrack songs synthesized from MIDI with professional virtual instruments.The study uses the SMG baseline’s train, validation, and test splits.Source: A. Dataset · p. 3
  • The MIDI source provides exact tempo maps and time signatures, supplying a noise-free metrical grid for the study.Real-audio recordings lack such an exact grid, so tracker-derived annotations are deferred to future live-tracked evaluation.Source: A. Dataset · p. 3
  • Each example is a ten-second window with one stem held out as the target and the other stems mixed as input.Windows contain 500 frames at 50 Hz.Source: A. Dataset · p. 3

B. Setup and Evaluation Protocols

The experiments use oracle dataset grids to measure performance with perfect conditioning information. Evaluation follows the baseline protocol and includes headline, metric-specific, and long-horizon assessments.

B. Setup and Evaluation Protocols

  • All conditioned variants receive the oracle dataset grid, isolating performance with perfect conditioning while leaving causal prediction of that signal to future work.Variants train for 200k steps with AdamW and cosine learning-rate decay.Source: B. Setup and Evaluation Protocols · p. 3
  • Table I reports headline results on the Slakh2100 test split, with fv = 0 strictly causal and fv = +50 representing one chunk of look-ahead.The table caption specifies mean ± standard deviation over five seeds and includes a held-out real-stem ground-truth row.Source: Table I · p. 4
  • Evaluation uses 1,024 held-out windows, continued chunk by chunk at k = 50 with fv = 0 unless stated otherwise, and averages five sampling seeds.The protocol reports mean and standard deviation across seeds.Source: B. Setup and Evaluation Protocols · p. 4
  • Beat-F measures pulse agreement between generated accompaniment and input, CoCoLa scores harmonic and percussive coherence, and FAD measures distributional audio quality.The long-horizon protocol generates twenty-second continuations and scores the two ten-second halves separately.Source: B. Setup and Evaluation Protocols · p. 4

C. Results and Ablations

Metrical conditioning substantially improves beat alignment over the causal baseline and the non-causal reference, while auxiliary heads further refine alignment and coherence. The phase channels are central to the alignment gain.

C. Results and Ablations

  • Beat-F rises from 0.133 for the causal baseline to 0.432 with metrical conditioning and auxiliary heads, 3.2× the baseline and above the non-causal reference’s 0.269.The full model’s CoCoLa score is 60.84, within one point of the non-causal reference’s 61.70.Source: C. Results and Ablations · p. 4
  • CoCoLa rises from 58.56 to 60.03 with metrical conditioning alone, and FAD improves from 5.56 for the baseline to 4.25 with SiMe alone.Adding pitch and spectrum heads yields a best FAD of 3.96, while the future-token head trades some distributional fidelity for further alignment and coherence gains.Source: C. Results and Ablations · p. 4
  • Without metrical phase, Beat-F remains 0.141, while auxiliary heads alone leave it at the 0.133 baseline, identifying phase as the core alignment mechanism.Metrical conditioning alone reaches 0.380 Beat-F; pitch and spectrum heads raise alignment to 0.411 before the future-token head’s additional gain.Source: C. Results and Ablations · p. 4
  • The headline model exceeds the non-causal reference’s Beat-F on drum, bass, and harmonic targets: 0.729 ± 0.012, 0.571 ± 0.027, and 0.363 ± 0.010, respectively.The conditioned models’ relative Beat-F declines are 3% for SiMe and 5% for the full model over the harder second half, versus 14–16% for unconditioned rows.Source: C. Results and Ablations · p. 4

D. Discussion and Future Work

The discussion identifies oracle-grid alignment and untested live-partner drift as boundaries for evaluation, while noting the value of shared clocks in live settings. The conclusion reports improved beat alignment under causal operation, including beyond the training horizon.

D. Discussion and Future Work

  • With a synchronized grid premise, the study does not test how the model responds when a live partner drifts from the set rhythm.The authors call for training and testing on real-audio recordings that are not fundamentally aligned to the metrical grid.Source: D. Discussion and Future Work · p. 4
  • Percussive accompaniment profits most from the metrical signal, while harmonic targets leave the largest share unrecovered; a chord-chart channel is proposed as a possible extension.These observations match the authors’ informal listening.Source: D. Discussion and Future Work · p. 4
  • A shared clock is described as cheap and drift-free for DAW, studio, and stage settings; without one, the grid must be estimated causally from incoming audio.The authors also identify tracker-derived conditioning and delayed-observation settings as next evaluation steps.Source: D. Discussion and Future Work · p. 4

V. CONCLUSION

  • Under strictly causal operation, Silent Metronome triples beat alignment over the baseline, surpasses a non-causal reference with one second of look-ahead, and retains its advantage beyond the training horizon.The paper describes the metrical signal as independent of generated audio and therefore unable to drift.Source: §V · p. 4
Loading 2609.07688v1…