Source-linked AI summary
Disentangled Skill Representations for Predictive Human Modeling
Mariah Schrum, Deepak Gopinath, Srijan Srivatsa, Guy Rosman, Tiffany Chen
TL;DR
AI systems need to infer human skill from behavior over time because skill is persistent, compositional, and distinct from transient performance. SAIL learns participant-level embeddings that blend novice and expert bases and use counterfactual subskill supervision for interpretable representation. Across racing and baseball, it achieves strong predictive performance and improves behaviorally grounded disentanglement and downstream instructor-feedback prediction.
Problem
Human skill is a persistent, compositional, behaviorally grounded construct that cannot be reliably inferred from single actions or outcomes.
Method
SAIL learns persistent participant-level skill embeddings using novice–expert basis blending and counterfactual subskill supervision.
Results
SAIL achieves strong predictive performance across racing and baseball, improves behaviorally grounded disentanglement, and improves downstream instructor-feedback prediction by 10%.
Takeaways & Limitations
SAIL provides stable, interpretable skill representations that support behavior prediction and downstream personalization tasks.
Takeaways & Limitations
The evaluation is limited by dataset scale and scope, especially the small, augmented baseball dataset, and relies on noisy predefined subskill metrics and a smooth novice–expert continuum.
Abstract
from arXiv · showhide
Understanding human skill is important for AI systems that collaborate with, coach, or assist people. Unlike typical latent variable estimation problems which rely on single observations, skill is a persistent, compositional, and behaviorally grounded construct that must be inferred from patterns over time. We introduce Skill Abstraction with Interpretable Latents (SAIL), a method for modeling human skill as an interpretable, multi-dimensional construct inferred from naturalistic behavior. Our approach produces a skill embedding that is robust to transient performance fluctuations and learns a transferable representation of human subskills. Furthermore, SAIL supports skill-informed behavior prediction that generalizes across a variety of in-domain contexts. We represent each individual with a persistent skill embedding that controls a blend between expert and novice bases and is trained using counterfactual subskill swaps for disentanglement. This design encourages representations that are both robust to performance variation and structured for interpretability. We demonstrate across racing and baseball that SAIL achieves strong predictive performance and consistently improves behaviorally grounded disentanglement over the evaluated baselines, while also improving downstream AI coaching performance.
1 Introduction
Human skill is a persistent, behaviorally grounded construct that AI systems must infer across repeated interactions to personalize support and anticipate behavior. SAIL addresses this need with stable, predictive, and interpretable participant-level representations.
- AI systems need human-skill models to personalize instruction, anticipate behavior, and adapt assistance over time.
- Human skill is persistent, compositional, and behaviorally grounded, requiring inference from repeated patterns rather than individual actions or outcomes.
- Skill differs from trial-specific performance because stable abilities generalize across contexts while performance reflects factors such as fatigue or risk-taking.
- The proposed representation targets construct validity, predictive utility, and interpretability through stability, cross-context prediction, and disentangled subskills.
- SAIL combines participant embeddings, novice–expert basis blending, and counterfactual supervision to learn stable, predictive, and interpretable skill representations.
- 10% improvement in downstream instructor-feedback prediction is reported across racing and baseball experiments.
2 Related Work
Prior work models latent behavior, reusable action policies, or participant-specific characteristics, but generally does not explicitly represent human skill as persistent, interpretable, and participant-level. SAIL combines participant embeddings, novice–expert basis blending, and counterfactual supervision to target that gap.
- Human skill must be inferred over accumulated behavior, whereas traditional measures such as completion time or accuracy are context dependent.
- Trajectory-based and reinforcement-learning methods commonly infer latent structure or represent skills as reusable action primitives and control policies.
- Autoencoders, variational autoencoders, contrastive learning, and participant-level representations encode behavior or persistent individual characteristics.
- SAIL differs by explicitly optimizing human skill representations for construct validity, predictive utility, and interpretable subskill decomposition.
3 Approach
SAIL learns persistent participant-level skill embeddings from repeated behavioral trajectories, separating stable skill from transient performance variation. It predicts behavior through novice–expert basis blending and uses behaviorally grounded metrics plus counterfactual swaps to structure subskill representations.
- 3.1 Participant-Specific Skill Embedding: The embedding is intended to remain stable across trajectories and transferable across observed in-domain contexts while decomposing into interpretable subskills.Skill metrics provide noisy behavioral proxies that supervise these structured components.
- 3.1 Participant-Specific Skill Embedding: SAIL assigns one learnable embedding to each participant and shares it across that participant’s trajectories to capture stable behavioral tendencies.This pools evidence across repeated observations rather than estimating skill independently from each trial.
- 3.2 Skill Representation via Novice–Expert Basis Blending: SAIL predicts behavior relative to canonical novice and expert bases, reducing the need for the embedding to encode fatigue, noise, environmental variation, and idiosyncratic execution.Basis trajectories explain common behavioral structure, while the embedding captures deviations associated with expertise.
- 3.2 Skill Representation via Novice–Expert Basis Blending: The participant embedding produces simplex-constrained basis weights and interpolation coefficients for an element-wise convex blend of novice and expert trajectories.Multiple bases and independently modulated subskills allow nonlinear skill representations rather than a single linear expertise axis.
- 3.3 Counterfactual Training for Subskill Disentanglement: Counterfactual subskill swaps replace one latent slice with a donor’s slice and require behavior and metrics to reflect that isolated intervention.This encourages each latent slice to encode its designated subskill through behavior rather than imposing constraints only on the latent space.
4 Domains and Datasets
SAIL is evaluated in racing and baseball batting, two domains involving coordinated subskills and measurable behavioral outcomes. Racing provides the richer primary dataset, while baseball serves as a smaller, narrower, synthetically augmented supplemental domain.
- 4 Domains and Datasets: SAIL is evaluated in high-performance racing and baseball batting, which involve coordinated mastery of multiple interacting subskills and measurable behavioral outcomes.The two domains have substantially different movement dynamics and subskill structures.
- 4.1 High-Performance Racing: The racing dataset includes six coach-identified subskills: vehicle handling, gaze control, know-how, control inputs, physical ability, and perceptual ability.These subskills correspond to professional coaching diagnoses and training interventions.
- 4.1 High-Performance Racing: Racing trajectories contain vehicle pose, speed, and control signals downsampled to 100 points per track segment, collected from 95 simulator participants spanning novices to experts.The dataset used racetrack as the task context and included two collection phases.
- 4.2 Baseball Hitting: Baseball hitting data came from 13 experienced players on a competitive adult team, with tee and machine-pitch batting contexts and coach-associated subskill metrics.The metrics cover the kinematic chain, pelvis pausing, and thigh pausing.
- 4.2 Baseball Hitting: The baseball dataset is smaller, narrower in subskill coverage, and augmented with synthetic trials, so it is treated as a supplemental secondary domain.Augmentations used time warping, noise injection, scaling, and estimated cross-condition offsets.
5 Results
SAIL is evaluated against representation-learning baselines and ablations on construct validity, predictive utility, disentanglement, and downstream coaching prediction across racing and baseball.
- Construct Validity: SAIL achieves strong construct validity across racing and baseball, with test–retest similarity of 0.995 in racing and 1.000 in baseball.Removing the novice–expert basis reduces clustering quality, while the no-CF ablation performs similarly on construct validity.
- Predictive Utility: SAIL achieves the best predictive performance in racing and remains competitive in baseball.Removing basis blending nearly doubles racing prediction error, whereas removing CF supervision has little effect on prediction.
- Disentanglement and Interpretability: SAIL consistently achieves the strongest disentanglement across racing and baseball.Removing CF supervision substantially reduces interpretability metrics, while AE-LC underperforms despite explicit metric supervision.
- Skill-Informed Coaching: 10.0% relative improvement in weighted F1 over the unconditioned model is achieved when predicting instructor feedback with SAIL embeddings.SAIL also significantly outperforms trial-time conditioning on weighted F1 and accuracy and improves macro F1 over no conditioning.
- External Validation by Professional Coach: Spearman ρ = 0.81, p < .001, 95% bootstrap CI [0.56, 0.94], n = 22, measures agreement between SAIL skill estimates and independent professional-coach ratings.The ratings were collected independently of the drill-based metrics used to train the representation.
6 Limitations
The evaluation is constrained by dataset scale and scope, noisy predefined subskill metrics, and the assumption of a smooth novice–expert continuum.
- Limitations: SAIL is evaluated on limited-scale data, particularly small and augmented baseball data, and depends on potentially incomplete or biased predefined subskill metrics.The method also assumes a smooth novice–expert continuum and does not establish a unique or complete computational definition of human skill.