Source-linked AI summary
TimeCues Studio: A Workspace for Music Annotation and Algorithm Prototyping
Sapir Caduri, Yoav Goldberg
TL;DR
Precise, varied music annotations are needed for multimedia synchronization, but existing tools lack corpus-scale collaboration and integrated algorithm development. TimeCues Studio unifies annotation, ambiguity-aware evaluation, algorithm comparison, and Python-based prototyping in one workspace. It also provides feature-rich, grid-locked visualization and supports a 109-track EDM corpus case study with structured boundary labels.
Problem
Precise synchronization needs labeled corpora containing coarse boundaries and fine-grained music-event markers, while existing tools lack corpus-scale collaboration and integrated evaluation.
Method
TimeCues Studio combines corpus annotation, team workflows, ambiguity-aware labeling, algorithm comparison, and Custom Detector prototyping in one workspace.
Results
A 109-track EDM corpus contained a median of about ten boundaries per track, with 20% flagged optional and the remainder critical.
Takeaways & Limitations
The workspace supports structured, ambiguity-aware annotation and evaluation across music corpora while keeping algorithm suggestions subject to human verification.
Takeaways & Limitations
Existing scoring approaches described by the authors use single timestamps or fixed tolerance windows and do not gracefully handle discrete multi-candidate boundaries.
Abstract
from arXiv · showhide
Multimedia applications require precise music annotation-labeled positions, segments, or loops-placed by hand or algorithmically. Machine-learning algorithms are scalable and effective but need annotated training data, scarce for many tasks. TimeCues Studio is an open-source workspace where algorithm-development teams annotate a music corpus, compare detection algorithms against those annotations, and prototype new ones. Unlike existing tools built for a single track at a time, TimeCues targets teams annotating whole collections, tightly integrated with algorithm development. Annotators place several marker types-each supporting ambiguity-aware labeling-on a grid-locked timeline that visualizes many music features, including separated audio stems. The same timeline drives an algorithm-comparison engine with bundled baselines, a Python sandbox for prototyping new models, and an ambiguity-aware evaluator that honors the structured fields. The same visualization suits solo annotators on music-sync projects. TimeCues is MIT-licensed and deploys via one Docker Compose command.
1 Introduction
TimeCues Studio addresses the need for precise, ambiguity-aware music annotations and integrated evaluation by combining corpus annotation, team workflows, algorithm comparison, and prototyping in one workspace.
- Motivation: Precise synchronization requires coarse segment boundaries and fine-grained markers for events such as snare hits, transients, and vocal entries.These annotations support music-driven lighting, video, and other multimedia synchronization tasks.
- Contribution: TimeCues unifies corpus annotation, team coordination, and algorithm evaluation, analysis, and comparison in a single open-source workspace.It targets the corpus-level and algorithm-development gaps of single-track annotation tools.
- Annotation: Annotators can use exact boundaries, single cues, overlapping spans, and DJ-oriented loops or patterns on a metric grid.The marker types support different music-analysis tasks.
- Ambiguity-aware annotations: The ambiguity-aware scheme records multiple valid candidates and credits predictions matching any candidate, including semantically equivalent markers grouped in layers.Critical and optional flags allow the dashboard to score important moments separately.
- Algorithm development: Custom Detectors, bundled baselines, and AutoGuess support model prototyping, side-by-side comparison, and semiautomated annotation suggestions.Custom Python scripts can inject hypotheses onto the timeline, while AutoGuess clusters predictions into candidate suggestions.
- Visualization: A shared visualization combines synchronized feature layers, beat-grid snapping, and per-stem rendering for annotation and algorithm inspection.The 3-band waveform separates bass, mids, and treble so band-level events can be located visually instead of through repeated listening.
- Deployment: TimeCues is MIT-licensed, extensible through configuration and custom taxonomies, and deployable with one Docker Compose command.A small CC0 demo corpus supports end-to-end trial without initially supplying audio.
2 Related Work
Related systems cover audio annotation, human-in-the-loop labeling, consensus, and music-driven show control, but TimeCues combines these strands around corpus-scale annotation and algorithm evaluation.
- Annotation software: Sonic Visualiser and Praat operate one track at a time, while other web tools focus on listening, pitch, tags, transcripts, or containerized verification.TimeCues extends this landscape with corpus, team, and algorithm-in-the-loop support.
- Human-in-the-loop, evaluation & consensus: TimeCues couples human-in-the-loop annotation with scheme-aware scoring and consensus clustering through its ambiguity-aware evaluator and AutoGuess.This builds on prior work addressing arbitrary single-reference timestamps and annotator disagreement.
- EDM & show control: EDM analysis and downstream applications such as stage lighting, sound-to-DMX, and synchronization systems motivate an authoring environment for precise music-linked events.The related work spans drops, peak structure, tempo, timbre, downbeats, and show-control consumers.
3 Annotation Lifecycle
The annotation lifecycle manages corpora and beat grids, supports layered and feature-rich annotation, and accelerates review with algorithm-generated suggestions while preserving human verification.
- Corpus management: Teams manage whole corpora with per-track progress, shared beat grids, and dashboards showing team progress and annotator agreement.An administrator prepares the corpus before reviewers annotate its tracks.
- Beat-grid preparation: The importer supports static, dynamic, and manual beat-grid modes for fixed tempos, drifting tempos, and live or stitched takes.Static mode offers estimator suggestions, dynamic mode uses an editable tempo curve, and manual mode places beats individually.
- Corpus management: The dataset page manages storage and imports or exports annotations and audio in formats including JSON, Audacity, Sonic Visualiser, JAMS, MIDI, and REAPER.The supported import and export formats differ.
- Annotator Tool: Annotations use layers containing cues, non-overlapping boundaries, overlapping spans, or repeated DJ-style loops and patterns on a grid-locked timeline.Each layer groups markers sharing a type and meaning.
- Annotator Tool: The annotator can zoom synchronized visualizations, replay selected regions, edit labels, move or trim regions, and snap markers to beats.These interactions support isolating events and aligning annotations with producer beats.
- Visualization: The default 3-band waveform separates bass, mids, and treble, while additional signals such as spectrogram, MFCC, and chroma can be stacked.The interface is designed to expose different signals for different analysis and annotation tasks.
- Visualization: Signals can be rendered for individual Demucs stems, including vocals, drums, bass, and other separated sources.This enables views such as vocal-only chroma or drum-only spectrograms.
- Assisted annotation: Custom Detectors and AutoGuess prefill annotation layers with algorithm suggestions that annotators can accept, reject, or copy into a human-verified gold layer.AutoGuess surfaces consensus clusters as review cards.
4 Model Development Lifecycle
TimeCues unifies corpus-scale model comparison and prototyping through bundled detectors, Custom Detectors, shared timelines, and scheme-aware evaluation. It also supports consensus-based annotation assistance and multi-audience workflows.
- Model comparison: Algorithm Inspect overlays bundled detectors from major music-structure-analysis families and lets researchers add new entrants through Custom Detectors.The bundled families include novelty curves, feature-based pipelines, tensor decompositions, change-point detection, and deep models on demixed audio.
- Model comparison: Two dashboards compare predictions against manual or AutoGuess references for single songs and aggregate per-song scores across corpora.Boundary detection is the default scope, while cues, spans, loops, and patterns are available optionally or through Custom Detectors.
- Evaluation: The scheme-aware evaluator credits predictions within tolerance of any candidate and weights optional references separately from critical ones.It reports weighted recall, Mean Nearest-Boundary Distance, and Critical Section Recall in addition to mir_eval metrics.
- Prototyping: A Custom Detector is a browser-authored Python script that can serve both as a ranked algorithm candidate and as an annotation assistant.Its predictions can be surfaced for accept/reject review in the Annotator Tool.
- Consensus assistance: AutoGuess clusters predictions from selected algorithms into review cards and tunes consensus configurations by sweeping clustering parameters and ranking them by F1 Score.The cards show how many algorithms agreed, while the consensus also functions as a tunable baseline.
- Applications: TimeCues targets researchers, annotation teams, and show engineers with tools for model comparison, coordination, agreement tracking, and timing-tolerance specification.The critical/optional and multi-candidate fields support events requiring different timing tolerances.
A Case Study: Annotating an EDM Corpus
The authors used TimeCues to annotate a 109-track EDM corpus and found recurring structural labels alongside widespread boundary ambiguity. Shared beat grids and disagreement ranking supported consolidation into reviewed annotations.
- Corpus: 109 tracks spanning diverse EDM subgenres were annotated, with a median of about ten boundaries per track.Drops comprised 39% of boundaries, buildups 23%, breakdowns 11%, intros and outros 10% each, bridges 5%, and silences 3%.
- Corpus: About 20% of boundaries were marked optional, while the remainder were critical beat-locked transitions.The criticality field was actively used to distinguish timing requirements.
- Annotation: Annotators often placed boundaries by sight on a frequency-colored three-band waveform, sometimes faster than by ear.Band-level EDM structure appeared as visible waveform shapes.
- Annotation: Annotators recorded multiple defensible boundary positions when transitions could be heard at either the outgoing section’s last bar or incoming section’s first.Each annotator layer remained separate while sharing the beat grid.
- Agreement: The Team Dashboard ranked tracks by inter-annotator disagreement so administrators could target and consolidate the least-agreed tracks.Agreement used tolerance-windowed boundary F1 with greedy one-to-one matching and label agreement.
B.1 Model-assisted annotation
Model-assisted annotation uses predictions and consensus suggestions to shift labeling toward verification while preserving human-verified references. The system’s service architecture and deployment profiles support this workflow across hardware environments.
- Model-assisted annotation: Custom Detector predictions can be requested on demand and placed on the canvas as editable pre-annotations for accept/reject review.Verified labels accumulate into a corpus for fine-tuning, especially when pretrained models underperform on under-represented material.
- Model-assisted annotation: AutoGuess presents consensus clusters from multiple algorithms as review cards that function as semiautomated annotation suggestions and a tunable baseline.The cards support a review workflow rather than replacing verification.
- System architecture: Backend capabilities run as separate services because audio and deep-learning libraries can require conflicting dependency versions.Isolated services can track upstreams independently and restart without taking down the workspace.
- Deployment: A multi-architecture Docker image and Compose profiles support amd64 and arm64 environments while allowing optional GPU, CPU, and experimental-model services.Profiles prevent first-time users from downloading CUDA or experimental weights they do not need.
B.4 Sandboxed Python detectors
Sandboxed Custom Detectors let researchers edit and run Python scripts in the Playground, with results returned to the shared canvas for both interactive testing and corpus evaluation.
- Sandboxed detectors: A Custom Detector is a short Python script edited in the browser-based Playground and executed with Run to place results on the canvas.The same script supports interactive single-song testing and batch evaluation across the corpus.
- Sandboxed detectors: Detector scripts run server-side in isolated child processes with memory, CPU, and wall-clock limits.These limits contain runaway loops and accidental allocations.
- Sandboxed detectors: The Playground can feed Custom Detector outputs to the inspector, the annotator, or both.This connects browser-based prototyping to both model analysis and annotation assistance.
B.5 Pre-computed feature cache
TimeCues caches expensive audio analyses and stems for reuse across signals, source selections, and algorithm configurations, while exposing corpus storage components and cleanup controls.
- The feature server caches expensive computations, including spectrograms, MFCC, chroma, tempogram, SSM, novelty, and Demucs stems.Caches use song-slug and feature-name keys with file-hash invalidation.
- Built-in baselines and Custom Detectors share cached features, so new configurations require detection and scoring rather than repeated feature extraction.
- Figure 8 separates the on-disk footprint of stems, analysis, raw MSAF output, BPM, and algorithm clusters, with regenerable caches cleared in one click.
- Signals can be toggled and rendered for the full mix or individual Demucs stems through the shared analysis interface.The interface exposes synchronized analysis layers and per-stem source selection.
- Annotator identity is attached to saved annotations through Google or a username/email sign-in, enabling later comparison of annotators’ layers.
B.6 Files over database
TimeCues stores annotations as separate JSON files organized by layer, annotator, and song, preventing overwrites while supporting inspection and cache cleanup.
- Annotations are stored as one JSON file per song, layer, and annotator.The path uses data/annotations/<layer>/<annotator>/<slug>.json.
- Per-annotator subdirectories prevent annotators from overwriting one another’s files.
- The annotator selected at sign-in supplies the annotator component of each annotation path.
- Separate category files let the storage panel account for each category and clear regenerable caches in one click.
B.7 Shared beat-locked canvas
TimeCues uses one beat-locked, snap-enabled canvas across dataset preparation, annotation, and algorithm inspection, while supporting interoperable exchange and local demonstration.
- A per-song beat grid drives every view, and optional snapping aligns manual boundaries and algorithm predictions on shared grid lines.
- The same grid and shared canvas toolbar serve Dataset Prep, the Annotator Tool, and Algorithm Inspect.The toolbar includes grid mode and snap controls.
- TimeCues exports JSON, JAMS, Audacity, Sonic Visualiser, MIDI, and REAPER, and imports JSON, JAMS, Audacity, and CSV.
- A bundled three-track CC0 demo runs client-side via localStorage without uploading audio, creating an account, or using server-side data.
- A deployment pipeline builds and ships the image for both architectures.