Source-linked AI summary
VocalCoachBench: Benchmarking Audio-Language Models on Expert Feedback for Singing
Hayeon Bang, Hounsu Kim, Wonil Kim, Juhan Nam
TL;DR
Existing audio-language benchmarks largely evaluate recognition and description, leaving expert-grounded vocal coaching feedback insufficiently tested. VocalCoachBench combines structured and claim-based evaluation across controlled and diverse singing scenarios. Models can compare performances and identify broad issue domains, but fine-grained issue identification and strict diagnosis alignment remain weak.
Problem
Existing audio-language benchmarks rarely test whether models can identify domain-specific problems in audio and provide grounded corrective feedback for improvement.
Method
VocalCoachBench combines same-song controlled comparison, diverse-song segment-grounded coaching, deterministic structured targets, and claim-based evaluation of diagnosis and correction.
Results
Models can compare performances and identify broad issue domains, but Top-3 fine-grained issue identification remains below the majority baseline and strict diagnosis hit rates stay below 7%.
Takeaways & Limitations
VocalCoachBench extends audio-language evaluation beyond description toward analytic, audio-grounded expert feedback for singing.
Takeaways & Limitations
The benchmark focuses on English solo singing, uses audio-only one-shot evaluation, and relies on expert-provided segments for segment-grounded assessment.
Abstract
from arXiv · showhide
Recent audio-language models are increasingly evaluated on recognizing, describing, and reasoning about audio, but expert-facing applications require a different capability: producing feedback that identifies problems and suggests corrective actions grounded in the input. We introduce VocalCoachBench, a benchmark for evaluating audio-language models on expert vocal coaching feedback for singing. VocalCoachBench contains 515 recordings annotated by 18 professional vocal trainers, yielding 1,056 expert submissions and 12,051 atomic coaching claims. It comprises a same-song subset for controlled comparison and a diverse-song subset for segment-grounded feedback across varied songs and recording conditions. To accommodate the open-ended nature of expert feedback, VocalCoachBench sep- arates deterministic structured targets from claim-based assessment of free-form diagnosis and corrective guidance. Human annotation analysis shows that expert agreement varies strongly with label granularity, motivating hierarchical structured metrics and claim-based evaluation of open-ended feedback. Experiments with 12 recent audio-language models reveal a consistent gap: while models can compare performances and identify broad issue domains in free-form feedback, Top-3 fine-grained issue-label identification remains below label-prior baselines and strict diagnosis alignment stays below 7%. To our knowledge, VocalCoachBench pro- vides the first public testbed for evaluating audio-grounded expert feedback for singing, moving audio-language evaluation beyond description toward analytic feedback.
1 Introduction
Existing audio-language benchmarks emphasize recognition and description, while vocal coaching requires identifying performance problems and giving grounded corrective feedback. VocalCoachBench addresses this gap with structured and claim-based evaluation designed for open-ended expert feedback.
- Research gap: Existing benchmarks rarely test whether models can identify domain-specific audio problems and produce grounded, improvement-oriented expert feedback.
- Motivation: Expert vocal coaching jointly considers pitch, rhythm, breath support, vocalization, and expression before recommending concrete corrective directions.
- Benchmark challenge: Expert feedback is inherently open-ended because trainers may prioritize different issues or recommend different corrective strategies for the same performance.
- Benchmark design: The benchmark separates deterministic structured targets from claim-based evaluation of natural-language coaching reviews rather than forcing a scalar score or unique gold response.
- Benchmark design: VocalCoachBench combines same-song controlled comparison with diverse-song coaching scenarios and segment-grounded feedback.
- Evaluation: Its evaluation suite includes controlled rankings, Top-3 issue-label prediction, segment-level classification, and claim-based diagnosis and correction assessment.
2 Related work
Prior work evaluates singing quality, musical performance feedback, or expert skill understanding, but existing resources do not explicitly benchmark diagnosis and correction for vocal coaching.
- Audio-language evaluation: Audio-language benchmarks primarily address general understanding tasks such as captioning, question answering, instruction following, and dialogue.
- Singing assessment: Singing-assessment research studies acoustic and perceptual dimensions including pitch, vibrato, rhythm, timbre, pronunciation, scores, and rankings.
- Singing assessment: Public singing-assessment datasets remain limited and often emphasize scores rather than coaching feedback.
- Expert feedback: Expert-feedback resources such as NeuroPiano, CROCUS, Expert-Novice, and LLaQo mainly target piano or instrumental performance.
- Research gap: Prior datasets cover singing assessment, expert feedback, or localized commentary but do not explicitly evaluate diagnosis and correction for vocal coaching.
3 VocalCoachBench construction
VocalCoachBench converts open-ended vocal coaching into structured and claim-level benchmark targets, using complementary same-song and diverse-song subsets. Its annotations show substantial scale, multi-factorial feedback, and agreement that improves with coarser representations.
- Design motivation: VocalCoachBench replaces compressed scalar ratings with issue identification, diagnosis claims, corrective guidance, controlled comparison, and segment-grounded feedback.Scalar ratings showed low agreement, with overall Krippendorff’s α approximately 0.323 and aspect-level α values approximately 0.171–0.393.
- Dataset construction: The benchmark combines same-song recordings for controlled comparison with diverse-song recordings spanning varied songs, styles, and vocal registers.Same-song recordings control lyrics, melody, difficulty, and song structure; diverse-song recordings complement this controlled setting with musical diversity.
- Annotation protocol: 18 professional vocal trainers annotated 515 recordings, producing 1,056 expert submissions, 4,443 feedback blocks, and 12,051 atomic claims.Each submission contains a median of 4 feedback blocks and 10 atomic claims.
- Dataset statistics and annotation behavior: Feedback blocks are multi-factorial: 68.5% receive two or more issue-category tags, while the subsets exhibit different Top-3 issue profiles.Same-song annotations emphasize vocalization, breath support, and pitch; diverse-song annotations include relatively more diction, expression, and rhythm.
- Quality control and agreement: Human agreement is stronger for parent categories and Top-3 sets than for fine-grained labels, while same-song rankings provide a usable relative signal.Fine-level F1@3 is 0.579, parent-level F1 is 0.771, pairwise ranking agreement is 77.5%, and Kendall’s τ is 0.550.
4 Evaluation suite
The evaluation suite separates reproducible structured tasks from claim-based assessment of free-form coaching. Structured tasks test ranking, issue selection, and localized classification, while open-ended evaluation separately measures diagnosis coverage and correction validity.
- Structured evaluation: Structured evaluation covers same-song triplet ranking, ranked Top-3 issue prediction, and segment-level issue classification.The suite uses constrained targets such as rankings, issue labels, and expert-consensus segments.
- Triplet ranking: Triplet ranking decomposes three performances into pairwise comparisons and reports pairwise accuracy, Kendall’s τ, exact permutation accuracy, and cycle rate.Each model preference is scored against the expert-induced preference.
- Top-3 issue prediction: Top-3 issue prediction compares ranked selections against every available expert annotation using fine-level F1@3 as its main metric.Parent-level F1, Top-1 accuracy, and rank-aware nDCG@3 provide additional views of alignment.
- Segment-level issue classification: Segment-level classification evaluates issue recognition on consensus segments formed from overlapping expert spans with IoU ≥0.3 and shared issue tags.The model receives cropped audio and predicts the main issue for each consensus segment.
- Open-ended coaching evaluation: Open-ended coaching is evaluated against atomic claims, separating diagnosis hit rate from correction validity rate.Diagnosis labels distinguish strict hits, coarse hits, misses, and contradictions; correction labels distinguish valid, weak, and invalid guidance.
5 Experiments
The experiments benchmark 12 audio-language models using task-specific structured prompts or simpler prompts for models that cannot reliably follow long schema-constrained instructions. Outputs are parsed and normalized before metric computation.
- Models: The benchmark evaluates 12 audio-language models spanning open-weight and closed-source systems.Six open-weight models use the main claim/objective protocol, two short-form audio models use the simple protocol, and four closed-source systems are included.
- Prompting protocol: Task-specific structured prompts target coaching claims, Top-3 issues, and segment classification, while short natural-language prompts support models unable to follow long schemas reliably.Models are queried deterministically when supported; Kimi-Audio uses the authors’ recommended sampling path.
- Output processing: Model outputs are parsed against required schemas or normalized to canonical taxonomy labels before evaluation.Free-form simple-protocol reviews are decomposed into diagnosis and correction claims before judge-based scoring.
6 Results
Results show a split pattern: models can compare performances, but fine-grained issue identification and precise coaching diagnosis remain difficult.
- Structured tasks: Direct pairwise ranking is feasible for several models, with Qwen2.5-Omni reaching 68.7% pairwise accuracy and Qwen3.5-Omni Plus reaching 74.5%.
- Structured tasks: 62.4% majority-baseline F1@3 remains higher than every model’s fine-level Top-3 F1@3 score.The best main-protocol and closed-source scores are 44.6% and 50.1%, respectively.
- Structured tasks: Segment-level issue classification remains difficult because MiMo-Audio matches the majority baseline by predicting vocalization for every segment.
- Open-ended coaching: Strict diagnosis hit rates stay below 7% across all models, while broader Hit rates range from 17–63%.Models often identify broad issue domains but miss expert-prioritized fine-grained problems.
- Open-ended coaching: Correction validity is higher for more capable models, but plausible corrections can accompany loose or generic diagnoses.
7 Conclusion
VocalCoachBench establishes an expert-annotated benchmark that combines structured and claim-based evaluation for grounded vocal coaching feedback. Its results expose limitations of current models and define important boundaries for interpreting the benchmark.
- Conclusion: VocalCoachBench combines same-song comparison, diverse-song segment-grounded coaching, structured issue targets, and claim-based evaluation.
- Conclusion: Current models can compare performances and identify broad issue domains, but remain weak at fine-grained issue identification and strict claim-level alignment.
- Conclusion: The benchmark provides a public testbed for evaluating grounded expert feedback beyond audio description.
- Limitations and future work: VocalCoachBench focuses on English solo singing, with the same-song subset limited to a single song.
- Limitations and future work: The benchmark is audio-only and oneshot, while segment-grounded evaluation assumes expert-provided segments rather than end-to-end discovery.
- Limitations and future work: Claim coverage may under-credit valid feedback outside the available expert claims, and outputs should not substitute for professional instruction.
B Score-derived triplet ranking diagnostic
The score-derived protocol independently elicits scalar quality scores and sorts them into triplet rankings, but score ties and weak agreement make it a diagnostic rather than the primary protocol.
- Protocol: Direct pairwise comparison is primary because the gold target is a relative judgment rather than an absolute quality score.
- Protocol: The score-derived protocol elicits 0–5 quality scores independently for each recording, then sorts them to induce a triplet ranking.
- Diagnostics: Pairwise accuracy, Kendall’s τ, exact accuracy, and tie rate diagnose agreement between induced and expert rankings.
- Results: Qwen2.5-Omni ties on 92.6% of triplets, while R1-AQA and Audio Flamingo 3 tie on every triplet.
- Results: The best score-derived result reaches 43.7% pairwise accuracy with Kendall’s τ = .302, below the 50% random baseline.
- Interpretation: Scalar score elicitation is therefore treated as an auxiliary diagnostic rather than the main triplet-ranking protocol.
E Atomic claim extraction
The benchmark decomposes expert-written vocal feedback into grounded atomic issue, correction, and strength claims, then validates extraction and judge labeling with targeted audits.
- Extraction design: The extraction pipeline decomposes existing expert feedback rather than generating new audio-based coaching judgments.It uses a fixed prompt with grounding constraints and preserves source-faithful meaning.
- Validation: The resulting claims are structured representations of human feedback rather than independent expert annotations because minimal normalization may restore subjects or link corrections.The released prompt, settings, intermediate outputs, and final claims support inspection of this process.
- Claim schema: Claims use three types—issue, correction, and strength—with corrections linked to the issue claims they address.Corrections must specify a concrete behavior, practice method, adjustment, or tool; vague advice is excluded.
- Normalization rules: Extraction rules split distinct issues and issue-targeted corrections while preserving causal statements and excluding unsupported additions.The prompt also prohibits adding interpretation, severity, or explanation not supported by the original text.
- Validation: A 100-block audit found low overall extraction error rates, with remaining errors mainly involving context-sensitive role assignment.The extracted text remained grounded in the original feedback even when diagnosis, correction, or strength roles were occasionally misread.
F.2 Correction judge validation
Correction judging uses three labels validated against professional review, while the evaluation suite combines structured audio prompts with diversity diagnostics to distinguish grounded feedback from repetitive templates.
- Correction judge validation: 79.2% exact agreement and Cohen’s κ = 0.687 were obtained between the LLM judge and vocal coach on 72 correction-level decisions.The main labels were valid, weak, and invalid.
- Correction judge validation: Most disagreements occurred between valid and weak, while invalid judgments were more stable, with 17 of 23 LLM-judged invalid cases confirmed by the expert.The valid–weak boundary reflects whether guidance is sufficiently specific or merely relevant but underspecified.
- Correction judge validation: The safety-oriented label was excluded from main correction metrics because cases were rare and boundary agreement was lower in the small audit.As an auxiliary binary diagnostic, it achieved 84.5% agreement and Cohen’s κ = 0.482 on 97 usable cases.
- Evaluation protocols: The evaluation suite uses separate coaching-claim, Top-3-plus-score, and structured-segment prompts, with deterministic decoding when supported.The structured tasks target diagnosis categories, quality scores, and segment-level issue classification.
- Response diversity: Diversity diagnostics are needed because repeated templates can inflate open-ended claim overlap without demonstrating audio-specific feedback.R1-AQA had 0.2% unique diagnosis and correction rates with Canon-5 at 100.0%, whereas high surface diversity alone was insufficient evidence of quality.
H.2 Annotator training, compensation, and quality control
Annotators received training before the main annotation phase, with staged calibration and daily workload limits intended to support consistency and reduce fatigue.
- Training and compensation: Annotators attended a training session covering task objectives and annotation guidelines before annotation began.The overall process lasted approximately seven days, and compensation was approximately USD 20 per hour.
- Quality control: Workload increased from two examples on day one to five on day two before the main annotation phase began on day three.Initial annotations were reviewed for consistency with the guidelines.
- Quality control: Annotators were limited to approximately 1.5 hours of work per day during the four-day main phase to help prevent fatigue.The fixed daily submission amount was used to maintain annotation quality.
I Compute resources
The study used one H100 GPU for open-weight inference, public APIs for closed-source models, and resumable batch jobs for processing and judging.
- Compute resources: Open-weight model inference ran one model at a time on a single NVIDIA H100 80GB HBM3 GPU using CUDA 12.8.The study did not train any models.
- Compute resources: Closed-source models were evaluated through public APIs, while parsing, normalization, and judging ran as resumable batch jobs.Checkpointing and resumption accommodated interruptions and variable API latency.
NeurIPS Paper Checklist
The checklist addresses reproducibility, experimental reporting, societal impact, limitations, and documentation for VocalCoachBench. It records affirmative support for reproducibility and asset documentation, while noting that statistical significance tests are not reported and scope limitations are discussed.
- Limitations: The limitations discussion identifies English solo-singing, a single-song same-song subset, audio-only evaluation, expert-provided segments, and reference-supported open-ended evaluation as scope boundaries.
- Experimental result reproducibility: The paper states that its benchmark metrics are reproducible from described tasks, metrics, prompting, inference, parsing, and judging procedures.Additional prompts and validation details are provided in the appendix.
- Open access to data and code: Benchmark annotations, evaluation scripts, prompt templates, and documentation are reported as sufficient to reproduce metrics, subject to audio-source access terms and closed-source model APIs.
- Experimental setting/details: The paper describes dataset construction, splits, task definitions, model routing, prompting protocols, deterministic decoding, parsing, and judge validation.
- Experiment statistical significance: The authors report deterministic benchmark and agreement metrics but do not report bootstrap confidence intervals or statistical significance tests for model comparisons.
- Asset documentation: The paper documents expert annotations, prompts, evaluation code, and documentation, including the annotation schema, taxonomy, examples, validation, and intended benchmark use.