Source-linked AI summary
Variable-Rate Harmonic-Percussive Time-Scale Modification with Real-Time Playback in Python
Sayema Lubis, Clark Peng, Jared Carreño, TJ Tsai
TL;DR
Open-source TSM tools largely target offline processing, while accompaniment applications need playback rates that can change in response to a live performer. The paper adapts harmonic-percussive TSM for real-time variable-rate playback in Python and finds that lookup-based approximations reduce runtime while approaching the baseline's perceptual quality.
Problem
Open-source TSM implementations are designed almost exclusively for offline use, while real-time options are few and less suited to modification and integration with Python research tools.
Source: §1 · p. 1Method
Results
45–50% total runtime reduction; HPS-Approx systems with β = 0.25 and below have win rates slightly above 50%, suggesting statistical noise.
Source: §4 · p. 7Takeaways & Limitations
Use HPS-TSM-Realtime when its runtime is adequate; when computation must be reduced, β = 0.25 offers near-indistinguishable quality, while β = 0.5 permits slight perceptual degradation.
Source: §4 · p. 8The reported profiling assumes 22050 Hz mono audio and a regime where large memory access is not an issue, and is not a comprehensive runtime evaluation.
Source: §4 · p. 8
Abstract
from arXiv · showhide
Time-scale modification (TSM) has a number of open-source implementations, but these are designed almost exclusively for offline use, in which a recording is processed at a fixed rate and written out ahead of time. Applications such as automatic musical accompaniment require a different setting, which we call variable-rate playback: the recording to be stretched is known in advance, but the playback rate is not, and must change continuously in response to a live performer. The few implementations that generate output in real time are written in C++ and optimized for speed rather than for ease of modification, experimentation, and integration with the primarily Python-based research ecosystem. This paper describes a Python implementation of the widely used harmonic-percussive TSM method for the variable-rate playback setting. Harmonic-percussive separation is performed offline as a preprocessing step on the known input recording, while synthesis and playback are carried out in real time with a time-scale factor that may change at every frame. We further propose a family of variants that reduce runtime by replacing the phase vocoder's analysis-stage FFT and instantaneous frequency calculations with lookups into precomputed tables. Subjective listening tests with 24 participants (1114 pairwise ratings) show that these approximations become perceptually indistinguishable from the exact implementation once the precomputed hop size is sufficiently small, while reducing total runtime by roughly half. We characterize the resulting tradeoffs among precomputation, runtime, memory, and perceptual quality to guide algorithm selection, and we release our implementation as an open-source package.
1 Introduction
Open-source TSM research tools largely target offline processing, while variable-rate playback is needed for applications that respond to live performers. The paper presents a Python implementation of harmonic-percussive TSM and approximations for real-time use.
1 Introduction
- The paper adapts a widely used harmonic-percussive TSM method for real-time playback with a continuously changing rate.The recording is known in advance; the rate can change in response to a live performer.Source: §1 · pp. 1–2
- The focal harmonic-percussive method processes percussive and harmonic components with OLA and phase-vocoder methods, respectively.The paper describes it as one of the most widely used methods in the MIR community.Source: §1 · pp. 1–2
- Existing open-source TSM tools are designed for offline use; the two real-time open-source options identified are C++ libraries that are harder to modify and integrate with Python research tools.The paper also notes concerns about the methods used by those libraries: one is not state-of-the-art, while aspects of the other are unpublished.Source: §1 · p. 2
- The work aims to make real-time TSM adaptable to changing playback rates and easy to modify and integrate within Python-based research software.A motivating application is playback of a prerecorded accompaniment adjusted to match a user's live playing.Source: §1 · p. 2
- The paper contributes a Python/numba real-time implementation, runtime-reducing variants, and listening experiments to characterize tradeoffs among perceptual quality, runtime, and memory.The stated aim is to guide algorithm selection for different hardware constraints.Source: §1 · p. 2
2 System Description
HPS-TSM-Realtime combines phase-vocoder processing of the harmonic component with overlap-add processing of the percussive component, producing playback in real time while allowing the time-scale factor to change dynamically. HPS-TSM-Approx replaces selected phase-vocoder calculations with precomputed table lookups to reduce computation, with the precomputed hop size controlling the tradeoff.
2 System Description
- The system offers exact and approximate real-time variants so users can choose an algorithm suited to their hardware constraints.HPS-TSM-Approx is presented as a family of lower-computation approximations to HPS-TSM-Realtime.Source: §2 · p. 2
2.1 HPS-TSM-Realtime
- HPS-TSM-Realtime separates the known input recording offline, processes its harmonic component with a phase vocoder and its percussive component with overlap-add, then mixes the outputs for playback.The time-scale factor can change dynamically during synthesis and playback; “real-time” does not include the offline separation step.Source: §2.1 · pp. 2–3
2.2 Real-time Phase Vocoder
- The phase vocoder locates each current analysis frame using a hop Ha = Hs αt, then computes STFT coefficients from the frame windowed with a Hann window.Hs is fixed at N/4; experiments used N = 2048 samples and a 22050 Hz sampling rate.Source: §2.2 · p. 3
2.3 Real-time OLA
- OLA uses shorter windows than the phase vocoder to reduce transient doubling, processing four frames per phase-vocoder frame while holding the time-scale factor constant across those four frames.The OLA window is 256 samples, and its analysis hop is set by the current time-scale factor.Source: §2.3 · p. 4
2.4 Updating Buffer
- Each iteration adds the phase-vocoder synthesis frame across the buffer and overlays four shorter OLA frames before flushing the portion that will no longer change to playback.The buffer length equals the phase-vocoder window; the OLA frames each modify only a portion of it.Source: §2.4 · p. 4
2.5 HPS-TSM-Approx
- HPS-TSM-Approx replaces the phase vocoder’s analysis FFT and instantaneous-frequency calculations with precomputed lookups, with the precomputed hop size controlling approximation quality and resource use.A hop size of 1 is exact but requires exorbitant memory and computation.Source: §2.5 · p. 5
3 Experimental Setup
The evaluation compares five precomputation settings against the unapproximated HPS-TSM-Realtime baseline through controlled, in-person pairwise listening tests. Trials use fixed, randomly selected time-scale factors and randomly selected GTZAN audio segments.
3 Experimental Setup
- Listeners compared five HPS-TSM-Approx settings, with precomputed hop sizes from 2048 to 128 samples, against the unapproximated HPS-TSM-Realtime baseline.The settings correspond to normalized hop sizes β = 1.0, 0.5, 0.25, 0.125, and 0.0625.Source: §3 · p. 5
- Each pairwise trial used the same music segment in both systems and a fixed time-scale factor sampled uniformly on a log scale between 0.5 and 2.Listeners could choose which system sounded better or indicate that both sounded the same.Source: §3 · p. 5
- The audio widgets did not permit dynamic time-scale changes during the listening tests, although their output was computed in real time.The study used fixed, randomly selected factors after preliminary tests found that participants largely stopped using an interactive slider.Source: §3 · pp. 5–6
- Trial audio came from randomly selected GTZAN recordings, using the first 15 seconds of each selected track.GTZAN contains 10 genres with 100 thirty-second recordings each.Source: §3 · p. 6
- Twenty-four college students provided 1114 ratings in in-person tests using the same environment, computing hardware, and headphones.The setup was chosen to control hardware, connection, speaker, and listening-environment differences.Source: §3 · p. 6
4 Results
Listening tests found that smaller-hop approximations approached the baseline’s perceptual quality, while table lookups substantially reduced runtime. The results characterize the tradeoffs among preprocessing, runtime, memory, and quality to guide algorithm selection.
Listening-test analysis
- Win rate counts system votes plus half the “Sounds the same” votes, while a one-sample t-test assesses whether the system is significantly worse than baseline.A small p value indicates significant inferiority to the baseline.Source: §4 · p. 6
Runtime
- 45–50% total runtime reduction for HPS-Approx accompanies an approximately 50% reduction in phase-vocoder runtime.The phase vocoder accounts for about 93% of baseline runtime; the approximation avoids analysis-stage FFTs while retaining synthesis iFFTs.Source: Figure 3 · p. 7, §4 · p. 7
Tradeoffs
- Table 2 compares preprocessing FFTs per generated second, runtime per generated second, lookup-table memory per original-audio second, and baseline-comparison win rate.The caption identifies the table as a summary of pre-computation, runtime, memory, and perceptual-quality tradeoffs.Source: Table 2 · p. 7, §4 · p. 8
Listening-test results
- 45.1% win rate and p = 0.057 for β = 0.5; β = 0.25 and below slightly exceed 50%, reaching the listening-test noise floor.The β = 0.5 result was not significant at the 0.05 level; the authors recommend β = 0.25 for most scenarios and β = 0.5 when slight degradation is acceptable.Source: §4 · p. 7
Runtime
- 150 times faster than real time on consumer-grade hardware was achieved for 22050 Hz mono data, in profiling experiments described as proof of concept.Runtime would be slower for stereo or higher sampling rates, and the profiling regime did not address large-memory-access costs.Source: §4 · p. 8
Tradeoffs
- β = 0.5 or β = 0.25 can reduce runtime by roughly half; β = 0.25 offers higher perceptual quality but requires more memory.The baseline is preferred when its runtime is adequate; the approximations have roughly similar runtimes across β values, while smaller β values require more preprocessing and memory.Source: §4 · p. 8
5 Conclusion
The conclusion presents a variable-rate adaptation of harmonic-percussive TSM, with offline separation and real-time synthesis, plus table-lookup approximations assessed for runtime, memory, and perceptual-quality tradeoffs.
5 Conclusion
- A known recording can be played in real time at a continuously changing rate, with harmonic-percussive separation performed offline and synthesis and playback carried out on the fly.Source: §5 · p. 8
- The proposed approximations replace the phase vocoder’s analysis-stage FFT and instantaneous-frequency calculations with lookups into precomputed tables.Source: §5 · p. 8
- Subjective listening tests characterize approximation quality, and the paper analyzes tradeoffs among runtime, memory, and perceptual quality.Source: §5 · p. 8
- The implementation aims to facilitate research on real-time TSM applications and more flexible, creative uses of existing offline methods.Source: §5 · p. 8