PaperlayerView exact revision on arXiv

Source-linked AI summary

TART: A Modular Tool for Technique-Aware Audio-to-Tablature Guitar Transcription

Akshaj Gupta, Hwi Joo Park, Andrea Guzman, Shamak Gowda, Samhita Konduri, Jiachen Lian, Robin Netzorg, Gopala Anumanchipalli

arXiv:2609.11904v1cs.LG

TL;DR

Guitar audio-to-tab transcription struggles with expressive techniques, string-fret ambiguity, and noisy recordings. TART addresses these problems with a modular four-stage pipeline and outperforms prior systems across four zero-shot benchmarks in audio-to-MIDI and string-fret transcription.

  • Problem

    Guitar transcription systems often miss expressive techniques, assign incorrect string-fret combinations, and generalize poorly to noisy recordings.

  • Method

    TART combines audio-to-MIDI transcription, nine-class technique classification, audio-conditioned T5 string-fret assignment, and automated tablature generation.

  • Results

    Across four zero-shot benchmarks, TART achieves 81.35% audio-to-MIDI F50 and AudioFret achieves 71.8% Tab F1, improving over prior baselines by 6.67 and 8.5 points, respectively.

  • Takeaways & Limitations

    TART jointly models pitch, expressive technique, and string-fret position to generate guitar tablature with technique annotations directly from audio.

  • Takeaways & Limitations

    The audio-to-MIDI stage misses unpitched percussion, technique classification assigns only one technique per note, and fixed-rhythmic quantization can misgroup chords.

Abstract

from arXiv · show

Automatic Music Transcription (AMT) for guitar remains limited by three challenges: existing systems often fail to capture expressive techniques such as slides, bends, and percussive hits; they often assign notes to incorrect string-fret combinations; and they are typically trained on clean recordings, limiting their generalization to noisy real-world audio. To address these challenges, we propose TART, a modular four-stage audio-to-tablature pipeline consisting of (1) an audio-to-MIDI transcription model, (2) an expressive technique classifier, (3) an audio-conditioned T5 encoder-decoder for string-fret assignment, and (4) an automated tablature generator. We evaluate TART in a zero-shot setting on GuitarSet, EGDB, and two augmented benchmarks, Noisy GuitarSet and Noisy EGDB. Averaged across these four benchmarks, TART achieves 81.35% audio-to-MIDI F50 (+6.67 points over the best prior baseline), 71.8% string-fret Tab F1 (+8.5 points over the best prior baseline), and 54.08% end-to-end Tab F1. To our knowledge, TART is the first framework to generate guitar tablature with both fingering and expressive technique annotations directly from guitar audio.

1. INTRODUCTION

Guitar audio-to-tab transcription remains difficult because systems miss expressive techniques, misassign string-fret positions, and generalize poorly beyond clean recordings. TART addresses these gaps with a modular four-stage pipeline from audio to performance-ready tablature.

  • Challenges: Guitar transcription systems often miss expressive techniques such as slides, bends, harmonics, and percussive hits.
  • Challenges: Pitch redundancy causes incorrect string-fret assignments that can produce tablature unlike how a guitarist would physically play.
  • Challenges: Models trained on small, professionally recorded datasets generalize poorly to noisy real-world guitar recordings.
  • Contribution: TART combines audio-to-MIDI transcription, nine-class technique recognition, audio-conditioned string-fret assignment, and MusicXML tablature generation.The pipeline uses a high-resolution CRNN, temporal CNN-BiLSTM, audio-conditioned T5 encoder-decoder, and automated score rendering.

2. PRIOR WORK

Prior guitar transcription work addresses audio-to-MIDI, expressive techniques, and fingering separately, but remains fragmented by limited data, incompatible labels, and weak cross-dataset generalization.

  • Audio-to-MIDI: Guitar transcription has lagged behind piano partly because paired audio-MIDI data are limited and guitar is polyphonic.
  • Expressive techniques: Existing technique datasets cover different subsets of techniques and use incompatible label schemes.
  • Expressive techniques: Prior technique classifiers typically use fixed-length feedforward features that discard temporal dynamics distinguishing similar techniques.
  • System context: Figure 1 depicts the four-stage combination that maps input audio to output tablature.
  • String-fret assignment: Audio-based fingering models lack explicit pitch supervision, while symbolic models translate pitch sequences into string-fret choices.

3. FIRST STAGE: AUDIO-TO-MIDI CONVERSION

TART’s first stage maps guitar waveforms to MIDI note events using a high-resolution CRNN adapted with diverse training data and guitar-specific noise augmentation. It is evaluated zero-shot across clean and noisy benchmarks, where TART achieves the strongest average F50.

  • Stage 1 objective: Stage 1 maps a 16 kHz audio waveform to MIDI note events containing pitch, onset, offset, and velocity.
  • Model: The adopted high-resolution CRNN consumes log-mel spectrograms and predicts onset, offset, activation, and velocity maps.
  • Training: Training pools four datasets spanning instrument domains and recording environments to reduce transfer problems from clean acoustic audio.
  • Robustness: A stochastic noise augmentor simulates realistic recording conditions while preserving onset alignment, and noisy benchmarks add room-impulse-response corruption.
  • Evaluation: Evaluation uses zero-shot F50 with ±50 ms onset and ±50 cents pitch tolerances on clean and noisy GuitarSet and EGDB.
  • 3.3 Evaluation: 81.35% average F50 gives TART a 6.67-point lead over Riley et al. across four zero-shot benchmarks.The benchmarks are GuitarSet, EGDB, Noisy GuitarSet, and Noisy EGDB.

4. SECOND STAGE: EXPRESSIVE TECHNIQUE CLASSIFICATION

TART’s second stage classifies each detected guitar note into one of nine expressive technique categories using audio features and a temporal CNN-BiLSTM. The classifier is trained on a unified multi-dataset taxonomy and substantially outperforms prior models under identical splits.

  • Technique taxonomy: The classifier assigns each note one of nine labels, including bend, hammer-on/pull-off, harmonics, slide, vibrato, and percussive techniques.The taxonomy also includes palm muting and picking/no technique.
  • Input representation: Each note’s audio segment is converted into a normalized 128 × 92 sequence combining MFCC, log-mel, and chroma features.Features are computed at a 23 ms hop and sequences are padded or truncated to approximately three seconds.
  • Architecture: The temporal CNN-BiLSTM uses two convolutional blocks followed by a bidirectional LSTM, classification head, and nine-class softmax.The model contains approximately 160,000 parameters.
  • Training data: The training set consolidates five public datasets under a common nine-class taxonomy with stratified train, validation, and test splits.Training uses class weighting to address imbalance and early stopping based on validation performance.
  • Evaluation: All classes exceed 95% recall, while vibrato is hardest at 89.4% F1 because it is primarily confused with bend.Percussion-style techniques reach 99.5% recall.
  • Evaluation: 95.9% Macro F1: the CNN-BiLSTM substantially outperforms prior models, which achieve 71.6% and 62.7%, while using 13× fewer parameters than Stefani et al.’s MLP.The comparison uses identical splits on the unified dataset.

5. THIRD STAGE: STRING AND FRET ASSIGNMENT

AudioFret assigns guitar notes to performer-consistent string-fret positions by combining symbolic MIDI context with per-note audio timbral cues. Its scaled, audio-conditioned architecture achieves the strongest reported oracle Tab F1.

  • 5.2 Training: AudioFret extends a T5-style sequence-to-sequence fingering model with a larger backbone and raw-audio timbral features alongside MIDI tokens.The larger model uses 256-dimensional representations, six layers, eight heads, and approximately 15 million parameters.
  • 5.2 Training: The audio encoder extracts 200 ms mel-spectrograms around note onsets and prepends their embeddings before the MIDI sequence for self-attention fusion.This design avoids explicit fusion hyperparameters while combining symbolic and acoustic information end-to-end.
  • 5.3 Post-processing: Inference uses beam search with grammar and reach constraints that enforce valid durations, prevent duplicate strings within chords, and limit fret spans to five.These constraints target syntactic validity and realistic hand reach during tablature generation.
  • 5.4 Evaluation: AudioFret is evaluated in an oracle setting using ground-truth MIDI events and compares against TabCNN, Fretting-Transformer, and model ablations.Tab F1 requires matching onset, pitch, and string assignment under the stated tolerances.
  • 5.3 Post-processing: AudioFret achieves the best overall Tab F1 of 71.8%, improving 8.5 points over the original Fretting-Transformer baseline.Scaling contributes +3.7 points, while adding audio conditioning contributes a further +4.8 points.

6. FOURTH STAGE: TABLATURE GENERATION

The fourth stage merges technique labels, string-fret assignments, and tempo into an annotated event stream, then quantizes and renders it as MusicXML tablature.

  • 6. FOURTH STAGE: TABLATURE GENERATION: Stage 4 combines technique-annotated events, string-fret assignments, and BeatNet tempo into a fully annotated stream before MusicXML rendering.The intermediate annotation is stored in a JAMS file and rendered as a beat-aligned tablature score.
  • 6. FOURTH STAGE: TABLATURE GENERATION: The generator clusters onsets within 30 ms, quantizes timing to a 1/16-note grid, groups shared onsets into chords, and resolves same-string collisions by duration.These operations provide the rhythmic structure used for score export.
  • 6. FOURTH STAGE: TABLATURE GENERATION: Single-note techniques are written directly, while hammer-ons, pull-offs, and slides are inferred between consecutive same-string notes within one beat.Users may provide capo, tuning, and tempo; otherwise standard tuning and no capo are assumed.

7. END-TO-END EVALUATION

End-to-end evaluation measures how errors in Stage 1 MIDI transcription affect downstream string-fret assignment and the complete tablature pipeline. Across four zero-shot datasets, the pipeline retains consistent performance on clean and noisy audio but incurs substantial propagation loss.

  • 7. END-TO-END EVALUATION: The evaluation compares oracle Tab F1 using ground-truth MIDI with end-to-end Tab F1 using Stage 1’s predicted MIDI.This isolates the downstream performance impact of imperfect audio-to-MIDI transcription.

8. CONCLUSION

TART is a four-stage system for generating guitar tablature that combines expressive technique labels with playable string-fret fingerings, and it outperforms prior systems across four zero-shot benchmarks. The authors also identify limitations in unpitched-note handling, multi-technique labeling, and rhythmic chord grouping.

  • 8. CONCLUSION: TART converts guitar audio into tablature with both expressive technique labels and playable string-fret fingerings, outperforming prior systems across four zero-shot benchmarks.The pipeline combines audio-to-MIDI transcription, technique classification, AudioFret, and tablature generation.
  • 8. CONCLUSION: The audio-to-MIDI stage misses unpitched percussion, preventing downstream technique annotation.This limitation constrains coverage of percussive events in the generated tablature.
  • 8. CONCLUSION: The technique classifier assigns only one technique per note, so simultaneous techniques such as bend with vibrato are not represented.The limitation concerns the expressive annotation stage rather than string-fret assignment.
  • 8. CONCLUSION: Fixed-rhythmic quantization for chord grouping can produce occasional misgroupings during tablature generation.The constraint arises in the final rendering process.
Loading 2609.11904v1…