Source-linked AI summary
Parameter-Efficient Multi-View Proficiency Estimation: From Discriminative Classification to Generative Feedback
Edoardo Bianchi, Antonio Liotta
TL;DR
Proficiency estimation must capture subtle execution differences distributed across temporal events and multiple views, while supporting interpretable applications such as coaching and rehabilitation. The paper discusses SkillFormer, PATS, and ProfVLM, which combine efficient fusion, temporal sampling, and conditional generation; ProfVLM reaches 48.2% top-1 accuracy on Ego+Exos with 5.3M trainable parameters.
Problem
Proficiency estimation requires modeling subtle differences in timing, balance, body mechanics, and execution across multiple views and short temporal events.
Method
The paper discusses SkillFormer for selective multi-view fusion, PATS for proficiency-aware temporal sampling, and ProfVLM for conditional generation of labels and expert feedback.
Results
48.2% top-1 accuracy on Ego+Exos is achieved by ProfVLM with 5.3M trainable parameters, surpassing SkillFormer and using about 5× fewer trainable parameters than SkillFormer and 20× fewer than TimeSformer baselines.
Takeaways & Limitations
The results support combining selective view fusion, proficiency-aware temporal sampling, and generative expert-style feedback in efficient multi-view systems.
Takeaways & Limitations
Per-domain results remain heterogeneous, with no single temporal configuration optimal for all activities.
Abstract
from arXiv · showhide
Estimating how well a person performs an action, rather than which action is performed, is central to coaching, rehabilitation, and talent identification. This task is challenging because proficiency is encoded in subtle differences in timing, balance, body mechanics, and execution, often distributed across multiple views and short temporal events. We discuss three recent contributions to multi-view proficiency estimation on Ego-Exo4D. SkillFormer introduces a parameter-efficient discriminative architecture for selective multi-view fusion; PATS improves temporal sampling by preserving locally dense excerpts of fundamental movements; and ProfVLM reformulates proficiency estimation as conditional language generation, producing both a proficiency label and expert-style feedback through a gated cross-view projector and a compact language backbone. Together, these methods achieve state-of-the-art accuracy on Ego-Exo4D with up to 20x fewer trainable parameters and up to 3x fewer training epochs than video-transformer baselines, while moving from closed-set classification toward interpretable feedback generation. These results highlight a shift toward efficient, multi-view systems that combine selective fusion, proficiency-aware sampling, and actionable generative feedback.
1. Introduction
Proficiency estimation assesses how well an action is performed, requiring fine-grained modeling of execution cues across time and camera views. The paper discusses SkillFormer, PATS, and ProfVLM as three Ego-Exo4D contributions addressing efficient fusion, temporal sampling, and feedback generation.
- Proficiency estimation models subtle differences in body mechanics, timing, balance, and movement consistency rather than action identity.
- These cues unfold over several seconds as micro-events that uniform sampling may miss and are best captured from multiple camera angles.
- SkillFormer provides a parameter-efficient multi-view discriminative architecture, PATS supplies architecture-agnostic temporal sampling, and ProfVLM generates proficiency labels with expert commentary.
2. Background and Related Works
Prior work moved action quality assessment from hand-crafted pipelines toward pretrained video encoders, multimodal fusion, and language-based interaction. Multi-view proficiency estimation now relies on synchronized complementary streams and compact, parameter-efficient modeling, while adaptive coaching remains largely unaddressed.
- Action quality assessment evolved from hand-crafted scoring pipelines to deep models built on pretrained video encoders.
- Expert-annotated datasets emphasize aligning synchronized views that provide complementary cues about body kinematics, object interactions, and the environment.
- Ego-Exo4D pairs an egocentric stream with up to four exocentric views and provides proficiency labels plus free-form expert commentary.
- Video transformers, instruction-tuned VLMs, compact language models, and LoRA support multi-view proficiency modeling and structured textual feedback.
- Coaching agents that adapt feedback across sessions remain largely unaddressed.
3. Methods
The methods combine multi-view data, selective fusion, proficiency-aware temporal sampling, and generative feedback. SkillFormer and ProfVLM use TimeSformer-based architectures, while PATS changes sampling without changing the model or training setup.
- Dataset: Ego-Exo4D provides synchronized egocentric and exocentric views across six skill domains, with four-level proficiency labels and expert commentary.
- SkillFormer: SkillFormer adapts a shared TimeSformer with LoRA and fuses view embeddings through CrossViewFusion using cross-view attention, gating, and adaptive self-calibration.Its reported configurations use 32 Ego frames, 24 Exo frames, and 16 Ego+Exos frames.
- PATS: PATS concentrates frames within short continuous action segments while still sampling multiple video parts, preserving locally dense movement excerpts.
- PATS: PATS improves all Ego-Exo4D view configurations, reaching 47.3% for Ego, 46.6% for Exos, and 48.0% for Ego+Exos.The largest gains occur in bouldering, music, and basketball.
- ProfVLM: ProfVLM replaces a classification head with conditional language generation, using a frozen TimeSformer, AGP fusion, and a LoRA-adapted language model to produce labels and feedback.
- ProfVLM: With 5.3M trainable parameters, 8 input frames, and 6 training epochs, ProfVLM reaches 48.2% top-1 accuracy on Ego+Exos.This surpasses SkillFormer while using about 5× fewer trainable parameters than SkillFormer and 20× fewer than TimeSformer baselines.
4. Discussion
The results identify selective view fusion, locally dense temporal sampling, generative output, and domain-aware adaptation as complementary design principles for multi-view proficiency estimation.
- Design lessons: The four design lessons are selective view fusion, temporal sampling, generative output, and domain-aware adaptation.These lessons summarize the results reported in Tables 1–3.
- View selection and fusion: 47.5% on Ego+Exos with 4.5× fewer trainable parameters shows that selective fusion improves over unstructured multi-view aggregation.The best TimeSformer Ego result is 46.8%, whereas Ego+Exos drops to 40.8%; SkillFormer reaches 47.5% with CrossViewFusion.
- Frames and temporal sampling: 8 frames suffice for ProfVLM’s best Ego+Exos result, while PATS improves SkillFormer across all view configurations by increasing local density within continuous segments.PATS obtains its largest gains in bouldering, music, and basketball, where structured fundamental movements are prominent.
- From classification to generation: 48.2% on Ego+Exos shows that ProfVLM’s gated view projector slightly surpasses SkillFormer+PATS while using roughly one fifth of the trainable parameters.ProfVLM replaces the classification head with a language model that emits a structured Level+Feedback response and also produces expert-style feedback without an accuracy penalty.
- Domain-aware adaptation: Per-domain heterogeneity means no single temporal configuration is optimal across activities, motivating domain-specific adapters or sampling policies.Domains differ in useful viewpoints, preferred sampling density, and required temporal continuity.
5. Conclusions and Outlook
The discussed methods advance Ego-Exo4D proficiency estimation with smaller trainable-parameter budgets and point toward interactive, structured, and privacy-aware systems.
- SkillFormer, PATS, and ProfVLM advance Ego-Exo4D proficiency estimation with substantially reduced trainable-parameter budgets.
- The methods shift the design space toward selective view fusion, smart temporal sampling, and generative expert-style feedback.
- Future systems could combine video-LLM orchestration, explicit pose cues, model compression, and benchmarks measuring feedback actionability, adaptation, personalisation, and privacy.
Declaration on Generative AI
The authors state that they did not employ Generative AI tools.
- The authors report no use of Generative AI tools.