Source-linked AI summary
ProfVLM: A lightweight video-language model for multi-view proficiency estimation
Edoardo Bianchi, Jacopo Staiano, Antonio Liotta
TL;DR
Existing proficiency-estimation approaches often predict labels or scores without explicitly modeling assessment reasoning. ProfVLM uses generative vision-language modeling to jointly produce proficiency labels and expert-like commentary from multi-view video, reporting improved accuracy with fewer parameters and less training time than comparison methods.
Problem
Many prior approaches predict proficiency labels or scores discriminatively, often from a single view, without unifying language-based reasoning and prediction.
Source: §1 · p. 2Method
Results
48.2% classification accuracy in Ego+Exos is reported alongside semantically aligned textual feedback, with ProfVLM using 5.3M trainable parameters and 6 training epochs.
Source: §7 · p. 26Takeaways & Limitations
Generated commentary provides actionable insights alongside quantitative proficiency evaluation scores.
Source: §1 · p. 3Training relies on proficiency labels and expert commentaries, which may be difficult to obtain at scale or may vary across annotators.
Source: §6 · p. 25
Abstract
from arXiv · showhide
Most existing approaches formulate action quality assessment and skill proficiency estimation as discriminative prediction tasks, typically producing discrete labels or scores without explicitly modeling the reasoning process underlying the assessment. We instead reformulate the problem as generative vision-language modeling, introducing ProfVLM, a parameter-efficient vision-language model that jointly predicts proficiency levels and generates expert-like natural language feedback from multi-view videos. ProfVLM leverages conditional language generation to provide actionable insights along with quantitative evaluation scores. Central to our method is an AttentiveGatedProjector that dynamically fuses and projects multi-view egocentric and exocentric features from a frozen TimeSformer backbone into a language model fine-tuned for feedback generation. Trained on EgoExo4D with expert commentaries, ProfVLM surpasses state-of-the-art methods while using up to 20x fewer parameters and reducing training time by up to 60% compared to existing classification-based methods. By providing natural language critiques aligned with performance levels, this work shows that generative vision-language modeling offers a powerful and efficient paradigm shift for interpretable action quality assessment.
1. Introduction
ProfVLM reframes proficiency estimation as conditional generation, producing proficiency labels and expert-like commentary from multi-view video. It addresses the need to align visual evidence with linguistic reasoning while aiming for efficient, interpretable assessment.
1. Introduction
- Assessing proficiency in complex activities matters for coaching and rehabilitation, and requires technical understanding beyond action recognition.The paper characterizes this understanding as challenging even for state-of-the-art architectures.Source: §1 · pp. 1–2
- Prior approaches commonly predict scores or labels discriminatively, often use single views, and do not unify proficiency prediction with language-based reasoning.Separate objectives for prediction and explanation limit alignment between visual evidence and linguistic reasoning.Source: §1 · p. 2
- ProfVLM treats proficiency estimation as conditional generation, jointly producing proficiency labels and expert-like explanatory commentary from multi-view video.The paper presents this as a generative alternative to classification-based proficiency estimation.Source: §1 · p. 2
- The model uses an AttentiveGatedProjector for structured multi-view fusion through cross-view attention and learnable gating, aligning video representations with the language-model embedding space.The architecture combines multi-view video input with a language model that generates proficiency labels and commentary.Source: §1 · p. 2, Figure 1 · p. 3
- ProfVLM improves multi-view accuracy using 20x fewer parameters (5.3M vs. 121M), 2x fewer frames (8 vs. 16), and 60% less training time (6 vs. 15 epochs).It also reports better accuracy than recently published works using 5x fewer parameters (5.3M vs. 27M) and 2-4x fewer frames (8 vs. 16/24/32).Source: §1 · p. 3
2. Background and Related Works
Prior work spans action quality assessment, multimodal and multi-view analysis, and video-language modeling. ProfVLM connects these areas by making proficiency prediction part of generative language modeling.
2. Background and Related Works
- Sports analysis applications span multi-view officiating, broadcast enhancement, and performance analytics, with interpretability and rapid feedback relevant to these settings.The related-work discussion also describes efficiency as important for sports analysis and skill assessment applications.Source: §2 · pp. 3, 4, 6
- Earlier action-quality and proficiency models use video regression or transformer-based prediction, while newer multimodal work combines cues such as RGB, depth, pose, and audio.The related-work discussion notes interpretability and domain-specific supervision limitations in earlier models, and highlights the value of egocentric and exocentric perspectives in recent benchmarks.Source: §2 · p. 4
- Prior multi-view and language-based assessment methods still use separate discriminative heads for proficiency labels, unlike ProfVLM’s unified autoregressive objective.Related work also describes parameter-efficient multi-view assessment and physiological-signal approaches, alongside domain-specific temporal sampling strategies.Source: §2 · p. 5
- Video-language models have advanced video-text alignment and comprehension through joint modeling, instruction tuning, temporal token compression, and decision-making approaches.The cited work ranges from VideoBERT and VideoLLaMA to recent models such as VALOR, SmolVLM, VideoAgent, and Apollo.Source: §2 · pp. 5–6
- ProfVLM bridges generative vision-language modeling and skill assessment by jointly predicting structured proficiency levels and generating expert-like feedback.The approach uses frozen video transformers and LoRA-tuned language models.Source: §2 · p. 6
3. Proposed Methodology
ProfVLM assesses synchronized multi-view video by generating both a proficiency label and commentary. Its method combines frozen video features with an attentive gated projector and a language model trained to produce structured responses.
3.1. Problem Formulation
- Each sample contains synchronized egocentric and up to four exocentric views, and the model must predict one of four proficiency categories and generate a performance comment.The categories are Novice, Early Expert, Intermediate Expert, and Late Expert.Source: §3.1 · p. 6
3.2. Video Feature Extraction
- A frozen TimeSformer extracts spatio-temporal representations; fixed 8-frame clips match its pretraining format, and synchronized views are processed while retaining view-specific features.The backbone is kept frozen, and the paper motivates its use by benchmark comparability and its divided space-time attention design.Source: §3.2 · p. 7
3.3. AttentiveGatedProjector: Multi-View Vision-Language Alignment
- The AttentiveGatedProjector normalizes views independently, uses cross-view attention to capture dependencies, gates fused features, and projects them into the language model’s embedding space.The projector is designed for generative video-language integration, including distribution matching and semantic coherence across autoregressive generation.Source: §3.3 · pp. 8–9
3.4. Tokenization and LLM Prompt Structure
- The prompt uses video delimiters and a chat format; training supervises system, user, and assistant messages, while inference supplies system and user messages for label-and-commentary generation.At convergence, both required output fields reach 100% schema compliance, learned during supervised fine-tuning even though the inference prompt does not enforce the template.Source: §3.4 · pp. 9–10, Table 1 · p. 10
4. Experimental Setup
The experiments use EgoExo4D’s multi-view proficiency data and expert commentaries, train ProfVLM with a compact video-language setup, and evaluate both classification and generated-language performance.
4.1. Dataset
- The six-domain benchmark pairs first-person video with up to four third-person views and provides proficiency labels plus expert commentary; evaluation uses official training and held-out validation splits.Each video-commentary pair is a training instance, expanding the training set while preserving the original validation set.Source: §4.1 · pp. 11–12
4.2. Implementation Details
- ProfVLM uses a frozen TimeSformer on eight sampled frames and fine-tunes SmolLM2-135M-Instruct with LoRA for six epochs.Experiments include Ego, Exos, and combined views, with the AttentiveGatedProjector and MLP projector variants.Source: §4.2 · pp. 12–13
4.3. Evaluation Metrics
- Evaluation reports accuracy for benchmark comparability and F1-score to provide a more robust measure under class imbalance.The evaluation framework covers both classification and natural-language generation.Source: §4.3 · pp. 13–14
5. Results
The results section evaluates ProfVLM on EgoExo4D and covers baseline selection, overall accuracy and efficiency, scenario-level accuracy, and generated-feedback quality.
5. Results
- The EgoExo4D evaluation examines baseline selection, overall performance and efficiency, per-scenario accuracy, and qualitative generated-feedback results.The evaluation concerns multimodal proficiency estimation.Source: §5 · p. 14
5.1. Baseline Selection Rationale
The authors focus comparisons on established multi-view proficiency methods because general-purpose VLMs lack task-specific multi-view fusion and would require substantial adaptation for a fair comparison.
5.1. Baseline Selection Rationale
- General-purpose VLMs lack specialized multi-view fusion, while adapting them would require architectural changes and task-specific training that complicate direct comparison.The authors therefore center comparisons on established multi-view proficiency methods.Source: §5.1 · pp. 14–15
- The zero-shot comparison tests PandaGPT, SmolVLM, and SmolVLM2 with basic frame concatenation; SmolVLM shares ProfVLM’s SmolLM2-135M language backbone.The shared backbone tests whether SmolLM2-135M alone can be competitive without ProfVLM’s specialized components.Source: §5.1 · pp. 15–16
- The tested general-purpose VLMs perform significantly worse, so the evaluation emphasizes multi-view methods SkillFormer and EgoPulseFormer.These methods represent the state of the art for this task on EgoExo4D.Source: §5.1 · p. 16
5.2. Overall Classification Accuracy and Efficiency
ProfVLM with AGP achieves the strongest reported combined-view accuracy while using fewer parameters, fewer frames, and fewer training epochs than prior approaches.
5.2. Overall Classification Accuracy and Efficiency
- 48.2% accuracy in the combined Ego+Exos setting lets ProfVLM (AGP) surpass EgoExo4D baselines, SkillFormer, and EgoPulseFormer under standardized splits.In that setting, AGP records 44.4% F1 versus 36.0% for MLP; accuracy is 48.2% versus 42.9%.Source: §5.2 · p. 16
- ProfVLM is more efficient, using 5.3M trainable parameters (a 5x reduction versus SkillFormer’s 14–27M) and 8 frames per video, compared with 16–32 frames in prior approaches.Source: §5.2 · p. 16
- 6 training epochs versus 15 for TimeSformer and EgoPulseFormer variants represents a 60% reduction in training time.AGP also consistently outperforms its MLP counterpart across view configurations.Source: §5.2 · pp. 16–17
5.3. Cross-Domain Accuracy Analysis
ProfVLM (AGP) leads five of six EgoExo4D scenarios, while view utility varies by activity and Basketball is the exception. The authors attribute Basketball’s weaker result to fine-grained spatial and temporal demands that the current design may not capture.
5.3. Cross-Domain Accuracy Analysis
- ProfVLM (AGP) achieves the highest accuracy in Dancing (60.35%), Bouldering (38.74%), Cooking (56.00%), Soccer (76.00%), and Music (72.05%).It does not lead in Basketball.Source: §5.3 · p. 17
- Ego+Exos improves on single views in Dancing (60.35% vs. 51.41%/53.8%), whereas exocentric input is strongest in Soccer (76.00%).Music performs equally well with single-view configurations; Dancing and Bouldering benefit from integrated views for spatial relationships and body positioning.Source: §5.3 · p. 17
- AGP consistently improves over the MLP projector, particularly in visually complex scenarios such as Dancing and Bouldering.The authors suggest global attention may suppress localized basketball cues, motivating domain-adaptive attention and higher-resolution temporal modeling.Source: §5.3 · pp. 17–18
- Basketball is the sole scenario where ProfVLM underperforms the baseline (41.00% vs. 55.24%).The authors suggest its fine-grained requirements, including precise hand actions and player-ball dynamics, may challenge current temporal sampling, spatial resolution, and global attention.Source: §5.3 · pp. 17–18
5.4. Ablation Studies
The ablations examine projector components, supervision quantity and modality, prompt ordering, and schema compliance. They report gains from the full AGP and multiple-commentary regime, label-first prompting, and continued fine-tuning.
5.4.1. Projector Architecture
- 12.3%: full AGP reaches 48.2% accuracy versus 42.9% for the MLP baseline.The progressive design adds cross-view multi-head attention, learnable gating, and adaptive normalization; the authors report each component contributes to the final performance.Source: §5.4.1 · p. 18, Table 6 · p. 19
5.4.2. Supervision Modality and Volume
- 48.2%: multiple-commentary training with 4,238 samples outperforms single-commentary training (40.7%) and label-only generation (45.0%).The authors attribute the full regime’s result to increased data volume and implicit regularization from diverse commentaries; single-commentary training is described as suffering loss imbalance and overfitting with 1,904 samples.Source: §5.4.2 · pp. 18–19, Table 7 · p. 20
5.4.3. Prompt Structure
- 2.9%: generating the label first yields 48.2% accuracy versus 45.3% when commentary comes first.The authors attribute the commentary-first drop to spurious dependencies on generated text and gradient dilution across 50–100 commentary tokens.Source: Table 8 · p. 20, §5.4.3 · pp. 20–21
5.4.4. Schema Compliance
- After one epoch, schema compliance reaches 100% for the label field and 96.7% for commentary; both reach 100% at convergence.Compliance checks JSON validity and the presence of the required “Proficiency Level” and “Proficiency Commentary” keys.Source: Table 9 · p. 21, §5.4.4 · p. 21
5.5. Language Generation Quality
ProfVLM generates feedback with semantic alignment to expert commentaries, though lexical overlap varies and outputs reflect the dataset’s assessment-oriented style. AGP strengthens visual grounding without changing the shared language backbone, and the model is designed for deployment efficiency.
5.5.1. Semantic Alignment and BERTScore Validity
- BERTScore F1 remains between 85.41 and 85.53 across configurations, indicating semantic alignment with expert commentaries.The authors report minimal variation across view and projector configurations and use RoBERTa-large to capture contextual similarity.Source: §5.5.1 · pp. 21–22, Table 10 · p. 22
- AGP improves prediction accuracy by 12.3% over MLP while both variants retain comparable commentary fluency through the same SmolLM2-135M backbone with LoRA.The authors attribute the difference to AGP’s cross-view attention and learnable gating for more discriminative visual features.Source: §5.5.1 · pp. 22–23
5.5.2. Lexical vs. Semantic Metrics
- METEOR (17.33-18.23) and ROUGE-L (14.47-15.67) are modest, while BERTScore remains relatively stable across quality levels (87.14-82.87).The authors report that lexical metrics degrade in lower-quality examples, from 36.22 to 5.25 for METEOR and from 20.22 to 7.59 for ROUGE-L, while feedback can remain conceptually aligned despite differing from reference wording.Source: §5.5.2 · pp. 23–24
5.5.3. Dataset-Driven Feedback Characteristics
- Generated feedback reflects EgoExo4D’s technical, assessment-oriented commentaries rather than detailed corrective instruction.The authors say the model learns the expert evaluation style and naturalistic speech patterns of uncurated audio transcriptions.Source: §5.5.3 · p. 24
5.5.4. Model Efficiency Considerations
- ProfVLM uses SmolLM2-135M with 5.3M trainable language-component parameters, and the authors report coherent commentary with semantic alignment to expert annotations.They present this design as a trade-off between quality and deployment efficiency for resource-constrained environments.Source: §5.5.4 · p. 24
6. Limitations
The authors identify constraints in supervision, evaluation, output format, view handling, and visual representation. They also note that keeping the video encoder frozen simplifies training but may limit domain adaptation.
6. Limitations
- Training assumes both proficiency labels and expert commentaries, which may be difficult to obtain at scale and may vary across annotators.Source: §6 · p. 25
- Automatic semantic-similarity metrics do not assess factual accuracy, pedagogical effectiveness, or actionability, which the authors say could be evaluated by experts.Source: §6 · p. 25
- Generating class labels as free-form text can produce misformatted or ambiguous outputs that complicate structured downstream use or reliability guarantees.Source: §6 · p. 25
- The current design assumes a fixed number of input views and does not yet adapt to variable camera counts or inconsistent frame counts.Source: §6 · p. 25
- The frozen TimeSformer encoder simplifies training and reduces computational cost but may limit domain adaptation, especially for fine-grained temporal understanding.Source: §6 · p. 25
- ProfVLM relies primarily on high-level visual features without explicitly modeling pose, body kinematics, or action-specific cues.The authors identify integrating structured pose and motion representations as future work.Source: §6 · p. 25
7. Discussion and Conclusions
ProfVLM combines proficiency estimation with generated feedback, reporting 48.2% accuracy in Ego+Exos and semantically aligned explanations. Its fusion results vary by activity, and Basketball remains challenging.
7. Discussion and Conclusions
- ProfVLM reports 48.2% accuracy in Ego+Exos, surpassing state-of-the-art classification accuracy while generating semantically aligned feedback.The evaluation also reports that ProfVLM outperforms video-only baselines and contemporary methods.Source: §7 · p. 26
- 12–23% gains in accuracy and F1 over MLP-based fusion appear in Dancing and Bouldering; exocentric views dominate Soccer, while combined views excel in spatial-reasoning activities.The reported view utility depends on the activity.Source: §7 · p. 26
- Basketball remains challenging across approaches, possibly because its technical details exceed the current temporal and spatial resolution capabilities.The authors identify these domain-specific challenges as future work.Source: §7 · p. 26
- Feedback achieves BERTScore >85 despite modest METEOR and ROUGE-L, and qualitative analysis finds coherent explanations across proficiency levels.The authors characterize the feedback as semantically aligned with expert annotations.Source: §7 · p. 26