Source-linked AI summary
MoveBench: A Rubric-Based Benchmark for Evaluating Large Language Models in Sports and Health Coaching
Yitong Li, Jiang Bian, Jingling Wang, Pan Ding, Chengying Gu
TL;DR
Sports and health coaching benchmarks have not adequately covered the action-oriented, personalized, and safety-sensitive demands of real-world coaching. MoveBench evaluates these demands using diverse coaching tasks and structured rubrics; across evaluated models, overall scores vary, while objective single-select performance is saturated.
Problem
Existing benchmarks emphasize knowledge, reasoning, or conversational safety, leaving limited evaluation of actionable, personalized coaching interventions under real-world constraints.
Source: §1 · p. 1, §2.1 · p. 2, §2.4 · p. 2Method
MoveBench evaluates 809 questions across PG, DA, and QA using exact-match MCQ scoring, three-judge weighted open-question assessment, and an item-level safety override.
Source: §6 · p. 10Results
Across 15 models, displayed overall scores range from 0.875 to 0.943, Claude 4.6 Opus is highest, and all models score 1.000 on successfully scored SS items.
Source: §6 · p. 10Takeaways & Limitations
MoveBench provides a structured basis for further evaluation, rather than certification of safe or effective coaching.
Source: §6 · p. 10Judge-model agreement with the expert-derived baseline is approximately 62%–76%, below the 80%–86% agreement among human experts.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Despite rapid advances in large language models (LLMs), their deployment in the sports and health domain remains high-risk due to the absence of domain-specific evaluation benchmarks. Existing general-purpose benchmarks (e.g., MMLU and HELM) primarily assess broad knowledge and reasoning capabilities, but fail to capture critical domain requirements such as exercise safety, physiological appropriateness, personalization, and actionable coaching quality. This gap is particularly concerning in fitness and health applications, where incorrect or unsafe recommendations can lead to physical harm. To address this limitation, we introduce MoveBench, the first comprehensive benchmarking framework specifically designed for evaluating LLMs in sports and health coaching. The proposed benchmark integrates heterogeneous task formats, including multiple-choice questions (MCQs) and open-ended question answering (OpenQA), grounded in real-world user scenarios and paired with structured rubrics emphasizing safety compliance, scientific validity, and training logic. Through large-scale evaluation of 15 diverse LLMs on 809 questions with 5,012 rubric-level criteria, our results reveal substantial performance variability, particularly in safety awareness, structured program design, and personalization capability, highlighting a critical gap between knowing and doing. Our work establishes a foundational evaluation standard for sports and health LLM applications and provides a reproducible framework for advancing safe, reliable, and context-aware AI-driven coaching systems. Open resources are available at https://github.com/SatoSakula/MoveBench.
1 Introduction
Existing benchmarks do not capture the action-oriented, personalized, safety-sensitive demands of sports and health coaching. MoveBench addresses this gap with real-world coaching tasks, multidimensional rubrics, and a reproducible evaluation pipeline.
1 Introduction
- Sports and health coaching requires evaluation of program design, physiological appropriateness, safety, and practical executability—not only isolated-answer correctness.Users seek structured, adaptive recommendations tailored to characteristics such as fitness level, injury history, and environmental constraints.Source: §1 · p. 1
- Existing benchmarks lack coaching-workflow tasks and fine-grained domain criteria, while single-metric schemes such as exact match miss the multidimensional quality of open-ended responses.Examples of coaching workflows include exercise prescription, training-plan generation, and performance-data interpretation.Source: §1 · p. 1
- MoveBench combines objective multiple-choice questions with realistic open-ended coaching scenarios assessed through rubrics for safety, scientific validity, personalization, coherence, and user experience.The evaluation framework also combines exact-match scoring with rubric-based scoring, weighted multi-judge aggregation, consistency checks, and a safety redline for harmful recommendations.Source: §1 · p. 1
- The benchmark’s contributions include diverse real-world task formulations, multidimensional rubric-based evaluation, and a scalable, reproducible pipeline with multi-judge aggregation.The stated contributions also position the framework for safety-critical applications.Source: §1 · pp. 1–2
2 Related Work
Prior benchmarks assess broad knowledge, medical or sports knowledge, and conversational quality, but leave a gap in evaluating personalized, executable, safety-sensitive coaching interventions. MoveBench frames this as an intervention-evaluation problem.
2.1 General-Purpose LLM Benchmarks
- General-purpose benchmarks assess knowledge and reasoning through exam-style or abstract tasks, rather than real-world domain-specific interventions with complexity, uncertainty, and safety constraints.Examples include MMLU [Hendrycks et al. 2021], MT-Bench [Zheng et al. 2023], and AlpacaEval.Source: §2.1 · p. 2
2.2 Medical and Health-Related Benchmarks
- Medical and health benchmarks cover exams, biomedical comprehension, and scenario-based conversations, but largely assess informational assistance or clinical reasoning rather than continuous personalized intervention.HealthBench uses multi-turn scenarios and physician-authored rubrics; action-oriented health applications remain under-covered.Source: §2.2 · p. 2
2.3 Sports and Fitness Domain Benchmarks
- SportQA [Xia et al. 2024] covers sports disciplines and rules, while fitness training, exercise prescription, rehabilitation, and personalized coaching workflows remain inadequately covered.Coaching evaluation additionally requires explicit treatment of user constraints, executable plans, and safety-sensitive recommendations.Source: §2.3 · p. 2
2.4 Gap: From Knowledge Evaluation to Intervention Evaluation
- Coaching interventions require actionable plans, personalization, temporal progression, and attention to physical safety; MoveBench evaluates training-plan generation, sport-data interpretation, and sports and health knowledge under coaching constraints.Its rubric-based approach complements existing general-purpose, medical, and sports-understanding evaluations.Source: §2.4 · pp. 2–3
3 Methodology
The methodology evaluates coaching across workout planning, sport-data analysis, and sport-and-health question answering using a generated, validated benchmark, task-specific rubrics, and weighted model judging with safety overrides. It also describes checks of scoring agreement and repeatability, including how attempted criteria, classified agreements, agreement counts, and missing or unclassified records are handled.
3.1.1 Personal Workout Plans.
- Workout-plan evaluation uses eight rubric dimensions spanning medical boundaries, plan structure and readability, program- and exercise-level adaptation, safety, and user constraints.Plans must synthesize user profiles, goals, environments, and injury conditions into structured, executable training programs; criteria draw on exercise science, human-computer interaction, and safety-critical design.Source: §3.1.1 · p. 3
3.1.2 Personal Sport Data Analysis.
- Sport-data analysis assesses whether models turn exercise logs and context into physiologically grounded insights and actionable recommendations.Its six dimensions cover data processing, analytical reasoning, expression, risk and safety, recommendation quality, and medical boundaries.Source: §3.1.2 · p. 3
3.1.3 Sport & Health Knowledge Question Answering.
- Sport-and-health QA evaluates scientific accuracy, teaching and expression quality, and safety-aware boundaries for ambiguous or partly incorrect user assumptions.The task emphasizes interpreting intent and guiding users toward safe, domain-relevant knowledge rather than assessing factual correctness alone.Source: §3.1.3 · p. 4
3.2 Benchmark Construction: Automatic Question Generation Pipeline
- Three complementary generation pipelines create QA, workout-plan, and data-analysis questions, with realistic profile sampling and metadata for subgroup analysis.QA prompts range from concise factual queries to context-rich questions; plans combine user constraints and use a safety rubric with a zero-score override when at least two of three judges flag a redline; data-analysis scenarios embed real activity logs and assess cross-metric reasoning and actionable recommendations.Source: §3.2 · pp. 4–5
3.3 Benchmark Construction: Question Quality Check Pipeline
- Automated quality checks screen question clarity, answer correctness, rubric consistency, scientific validity, and safety before domain experts approve, revise, or reject remaining items.Moderate- or high-risk items are automatically rejected; retained items must be evaluable, reproducible, and scientifically and safety aligned.Source: §3.3 · p. 5
3.4.1 Student Model Selection.
- The final student-model set consists of 15 models displayed in Figures 1 and 2, with figure labels serving as the reference naming convention throughout the manuscript.Source: §3.4.1 · p. 5
3.4.2 Judge Model Selection.
- Open-ended responses are scored by Gemini 3.1 Pro, Claude 4.6 Sonnet, and GPT-5.2 with weights 0.35, 0.35, and 0.30; two redline votes set an item’s earned score to zero.Objective items receive exact-match scores without partial credit; open-ended positive criterion scores are weighted across judges, and failures to obtain valid scores from all three judges remain unscored.Source: §3.4.2 · p. 5, §3.4.3 · pp. 5–6
3.5.1 Agreement and Repeatability.
- Agreement is defined as rubric-score differences of at most 0.2, while repeatability is summarized across three runs; a 30-question, 300-criterion expert-adjudicated subset validates judge comparisons.The tolerance is study-specific; agreement and repeatability are distinct, and neither alone establishes scientific correctness. Individual expert agreement with the adjudicated reference is reported separately from judge-model comparisons.Source: §3.5.1 · p. 6, §3.5.2 · p. 6
3.5.2 Validation Protocol.
- The protocol separates attempted criteria, criteria with reported agreement classifications, and agreement counts, while excluding missing or unclassified records from valid paired scores.Section 4.3 identifies recoverable count discrepancies and states the denominator used in each summary.Source: §3.5.2 · p. 6
4 Results
Across 809 questions and 5,012 rubric-level criteria, MoveBench results stratify the 15 models, with performance varying by coaching task and question format. Expert and judge scoring also show incomplete agreement and run-to-run variation.
4 Results
- The benchmark contains 809 questions and 5,012 rubric-level criteria, spanning 310 multiple-choice questions and 499 open questions.The capability-domain counts are 257 plan-generation, 291 data-analysis, and 261 question-answering questions; these are complementary views of the same benchmark.Source: §4 · p. 6
4.2 Question Type Performance Analysis
- Open-ended scores range from 0.812 for Gemini 4 Argon to 0.917 for Claude 4.6 Opus, while every model scores 1.000 on successfully scored single-select questions.Multi-select scores range from 0.888 to 0.980; strong structured-question performance does not necessarily carry over to open-ended coaching tasks.Source: §4.2 · p. 7
4.3.1 Expert Scoring Agreement.
- Expert agreement with the adjudicated reference ranges from 80.60% to 86.00% on classified pairs; one expert has 299 classified pairs rather than 300.The unresolved criterion is not assigned a cause or outcome.Source: §4.3.1 · p. 7, Table 1 · p. 7
4.1 Overall Performance Comparison
- Claude 4.6 Opus leads overall at 0.943; GPT-5.2 (0.929), GPT-5.5 (0.927), and Qwen3-Max (0.926) form a closely clustered top tier.Performance is stratified across models, and the authors caution that these results describe MoveBench rather than language-model capabilities generally.Source: §4.1 · pp. 6–7, Figure 1 · p. 8
4.3.2 Judge–Reference Alignment.
- Judge-model all-attempt agreement with the reference ranges from 62.33% to 76.33%, with Gemini highest; the authors report incomplete alignment.Claude's conditional rate is 193/280 = 68.93%; differing denominators and missingness preclude direct comparison with expert rates on classified pairs.Source: §4.3.2 · pp. 7–8, Table 2 · p. 8
4.3.3 Variation Across Repeated Judge Runs.
- Across three runs, Claude has the smallest observed sample SD (0.0472) and GPT the largest (0.1065), demonstrating run-to-run variation.With only three observations, these summaries provide limited evidence about future score distributions and cannot establish stable close rankings.Source: §4.3.3 · p. 8, Table 3 · p. 8
4.3.4 Judge Weight Selection.
- The stability-prioritized weight configuration has lower reported MSE (0.0177 versus 0.0183), while the balanced configuration caps the largest individual weight at 35%.The authors describe this as a committee-design trade-off, not evidence of statistical optimality.Source: §4.3.4 · pp. 8–9, Table 4 · p. 9
4.2 Question Type Performance Analysis
- Figure 2 separates model performance by single-select, multi-select, and open-ended formats, revealing different levels of variation across the task types.Single-select scores show a ceiling effect on the successfully scored subset; open-ended questions show the widest observed score range.Source: §4.2 · p. 7, Figure 2 · p. 9
5 Discussion
MoveBench assesses coaching across real-world tasks with a rubric-based framework, while its discussion highlights limits in judge–expert alignment and objective-task discrimination.
Discussion
- 809 questions and 5,000+ criteria support fine-grained assessment across objective and open-ended tasks through expert validation and multi-judge scoring.The benchmark covers personalized plan generation, sport data analysis, and health-related question answering.Source: §5 · p. 9
Limitations in Expert-Model Alignment
- Judge-model agreement with expert annotations is moderate, at approximately 62%–76% versus 80%–86% among human experts.Judges may diverge on nuanced reasoning, contextual judgment, and safety-sensitive rubric criteria, including borderline safety or personalization cases.Source: §5 · pp. 9–10
Student Model Score Differentiation
- All evaluated models score 1.000 on single-select tasks, while multi-select scores cluster around 0.88–0.98, limiting objective-task discrimination.The discussion characterizes this as saturation on knowledge-based and structured reasoning tasks under exact-match evaluation.Source: §5 · p. 10
Future Work
- Future work plans semiannual dataset updates, adding open-ended and long-context questions and expanding topics to rehabilitation, endurance training, and sport-specific coaching.Updates are planned to follow the generation–validation–evaluation pipeline.Source: §5 · p. 10
6 Conclusion
MoveBench evaluates sports and health coaching with 809 questions and 5,012 rubric-level criteria. Across 15 models, overall scores vary, while objective SS scores are saturated and OQ scores span a wider range.
6 Conclusion
- MoveBench contains 809 questions and 5,012 rubric-level criteria across PG, DA, and QA, combining exact-match MCQ scoring with weighted three-judge OQ assessment and an item-level safety override.The override requires at least two redline votes; Overall combines format scores by successfully scored item counts.Source: §6 · p. 10
- Overall scores range from 0.875 to 0.943 across 15 models, with Claude 4.6 Opus highest; successfully scored SS accuracy is 1.000 and OQ scores range from 0.812–0.917.The authors conclude that extended coaching responses should be evaluated separately from objective answer selection.Source: §6 · p. 10
- Judge–reference disagreement, run-to-run variation, and incomplete request coverage limit claims about exact rankings or real-world safety.MoveBench is presented as a structured basis for further evaluation, not as certification of safe or effective coaching.Source: §6 · p. 10
A Example Benchmark Questions and Evaluation Framework
The appendix presents three representative benchmark examples to show the range of task types and evaluation methods.
A Example Benchmark Questions and Evaluation Framework
- Three representative examples each include an original question, expected evaluation criteria, and detailed assessment rubrics.The examples are presented to demonstrate task diversity and evaluation methodology.Source: A Example Benchmark Questions and… · p. 11
A.1.1 Question Context.
The example describes a user seeking rapid weight loss before an event, with a limited preparation window.
A.1.1 Question Context.
- The user seeks to lose 2.5 kg in one week before a wedding, with approximately one month available.The profile frames a rapid weight-loss request under a short time constraint.Source: §A.1.1 · p. 11
A.1.2 Question.
The question asks whether liquid fasting is suitable for rapid fat loss before a wedding, with a proposed loss of 2.5 kg in one week.
A.1.2 Question.
- A person preparing for wedding photos next month asks about liquid fasting for rapid fat loss.The proposed method is limited to black coffee, juice, and milk each day.Source: §A.1.2 · p. 11
- The claimed target is losing 2.5 kg (about 5 pounds) in one week.Source: §A.1.2 · p. 11
- The question is whether this method is suitable for that short-term goal.Source: §A.1.2 · p. 11
A.1.3 Answer Choices.
The choices contrast claims about liquid fasting and supplements with a safety-focused recommendation against liquid fasting. The designated correct answer favors a moderate calorie deficit, high protein, and strength training.
A.1.3 Answer Choices.
- Choice A allows 3–5 days of liquid fasting for short-term weight loss, with hydration and a gradual return to normal eating.The next passage begins a contrasting choice B.Source: §A.1.3 · p. 11
- Choice B recommends L-carnitine with black coffee and claims two capsules daily directly convert fat into energy and enable easy fat loss without strict dieting.These are claims made by the answer choice, not endorsed guidance.Source: §A.1.3 · p. 11
- Choice C is correct: liquid fasting may cause muscle loss, metabolic slowdown, electrolyte imbalance, and cardiac arrhythmia; a safer alternative is a moderate deficit, high protein, and bodyweight strength training.The passage describes most fasting-related weight loss as water and muscle.Source: §A.1.3 · p. 11
- Choice D claims fasting reduces lactic acid and promotes fat oxidation, then proposes daily low-intensity cardio for “hypertrophy-style fat loss.”The supplied choice continues by claiming this improves muscle definition.Source: §A.1.3 · p. 11
A.1.4 Assessment Methodology.
These appendix examples illustrate exact-match MCQ scoring and a multi-criteria OpenQA rubric applied to user-specific coaching scenarios. They assess whether responses combine relevant training and nutrition knowledge with safety, structure, and personalization.
A.1.4 Assessment Methodology.
- A.1.4 assigns a single-select MCQ one point only for an exact match to answer C, with no partial credit.The rationale identifies the item as testing nutrition and exercise-science knowledge, including risks of extreme restriction and a safer alternative.Source: §A.1.4 · p. 11, §A.1.5 · pp. 11–12
A.2.1 Question Context.
- The A.2 scenario concerns a normal-BMI trainee whose chest growth has stalled during home training with limited equipment; the multi-select item requires both answers A and C for credit.The correct responses address progressive overload, nutrition, recovery, equipment constraints, and supplement safety; partial credit is not assigned.Source: §A.2.1 · p. 12, §A.2.4 · p. 12
A.2.5 Evaluation Rationale.
- The A.2 rationale tests integration across nutrition, training science, and supplement safety, while treating risky supplement claims and misconceptions about lactic acid as errors.It also identifies excessive fasted cardio and training to failure as problematic in the distractors.Source: §A.2.5 · p. 12
A.3.2 Question.
- The A.3 OpenQA scenario asks for a 6–10 minute K1 recovery walking–running plan for a woman with mild joint discomfort, a normal BMI, and a small outdoor space.The request specifies joint protection, very low intensity, and a stable heart rate.Source: §A.3.1 · p. 12, §A.3.2 · p. 12
A.3.3 Evaluation Framework.
- OpenQA responses are assessed with a multidimensional rubric; three illustrative criterion groups aid readability but do not replace the eight PG dimensions, and three judges aggregate scores and safety votes.At least two of three judges must flag a redline for the item score to be set to zero; one flag alone does not override the positive score.Source: §A.3.3 · pp. 12–13, §A.3.5 · p. 14
A.3.4 Rubric Dimensions and Criteria.
- A.3 safety criteria require conservative joint loading, K1 effort at perceived exertion ≤50%, heart rate near 100 bpm, stop conditions, and contraindication guidance.Medical overreach, unsafe advice, or invented physiology are redline violations; two judges must flag a redline to zero the item score.Source: §A.3.4 · pp. 13–14
- The A.3 rubric rewards structured delivery and training-goal alignment, including a warm-up, 4–7-minute main session, cooldown, gradual transitions, and low-impact exercise suited to the available space.Its criteria also require adaptation to the user's discomfort, normal BMI, space, and requested duration.Source: §A.3.4 · p. 13
A.3.5 Evaluation Rationale.
- The evaluation rationale treats actionable formatting, training-science competency, integrated safety, and respect for user constraints as distinct coaching capabilities.Ignoring the small-space or 6–10-minute constraints is described as generic rather than customized guidance.Source: §A.3.5 · p. 14