Source-linked AI summary
The Pros and Cons: Rank-aware Temporal Attention for Skill Determination in Long Videos
Hazel Doughty, Walterio Mayol-Cuevas, Dima Damen
TL;DR
Skill determination in long videos is difficult because relevant evidence is sparse and exhibited skill can vary across segments. The paper learns rank-aware temporal attention with separate high- and low-skill branches, achieving state-of-the-art performance across EPIC-Skills and BEST while introducing a larger five-task dataset.
Problem
Long videos contain irrelevant segments and can show differing skill levels across actions, challenging methods that assume uniform skill throughout the video.
Method
The model uses video-level supervision and a rank-aware loss to learn separate attention modules for segments indicative of higher and lower skill.
Results
The method outperforms previous work by 4.3% on EPIC-Skills and 5.4% on BEST, while outperforming uniform weighting and standard softmax attention.
Takeaways & Limitations
Rank-aware attention learns informative segments for skill ranking and achieves average performance above 80% on both evaluated datasets.
Takeaways & Limitations
Performance gains do not hold uniformly: Surgery is limited by I3D features, while Drawing and Origami decline when disparity loss is added.
Abstract
from arXiv · showhide
We present a new model to determine relative skill from long videos, through learnable temporal attention modules. Skill determination is formulated as a ranking problem, making it suitable for common and generic tasks. However, for long videos, parts of the video are irrelevant for assessing skill, and there may be variability in the skill exhibited throughout a video. We therefore propose a method which assesses the relative overall level of skill in a long video by attending to its skill-relevant parts. Our approach trains temporal attention modules, learned with only video-level supervision, using a novel rank-aware loss function. In addition to attending to task relevant video parts, our proposed loss jointly trains two attention modules to separately attend to video parts which are indicative of higher (pros) and lower (cons) skill. We evaluate our approach on the EPIC-Skills dataset and additionally annotate a larger dataset from YouTube videos for skill determination with five previously unexplored tasks. Our method outperforms previous approaches and classic softmax attention on both datasets by over 4% pairwise accuracy, and as much as 12% on individual tasks. We also demonstrate our model's ability to attend to rank-aware parts of the video.
1. Introduction
The paper frames skill determination in long videos as ranking subjects by selectively attending to skill-relevant temporal regions. It proposes rank-aware attention that separates segments indicating higher and lower skill and evaluates the approach on existing and newly collected datasets.
- Motivation: Skill determination assesses how well a subject performs a task, with video-based assessment aimed at leveraging everyday instructional videos.The motivation includes selecting which video a robot should imitate for tasks such as preparing scrambled eggs.
- Problem: Long-video methods may fail because they assume skill is consistent throughout the video, even though some segments are irrelevant and skill can vary across actions.For tying a tie, draping and straightening may be uninformative, while crossing the tie and pulling through the loop can reveal different abilities.
- Formulation: The paper treats skill determination as fine-grained video understanding, using temporal localization and video ranking rather than objective scores for common tasks.Crowdsourcing provides rankings through consensus when objective scores are difficult to articulate or certify.
- Approach: The proposed loss trains two attention modules to specialize in segments informative for high skill and sub-standard performance.These modules are fused to support overall skill assessment while preserving rank-specific attention.
- Contributions: The paper introduces rank-aware attention and a BEST dataset containing five YouTube tasks, 100 videos per task, 26 hours of video, and videos up to 10 minutes long.It evaluates the method alongside previous and attention-based baselines on EPIC-Skills and BEST.
2. Related Work
Prior skill-assessment work is limited by small or short datasets and approaches that use task-specific events, global features, or non-rank-aware attention. This paper instead learns rank-aware temporal attention for task-agnostic video ranking and evaluates it on longer videos.
- Skill Determination: Earlier video skill-assessment studies were limited despite demonstrating performance with task-specific features such as skeleton trajectories.The related work includes both task-specific and broadly applicable methods.
- Datasets: Existing datasets include short clips, small-scale collections, and a non-public figure-skating dataset, motivating evaluation on longer videos.The new dataset contains 500 videos across five daily-living tasks with average video length of 188 seconds.
- Long-Video Methods: Long-video skill assessment has used preselected events, global features, random segments, or figure-skating-specific recurrent attention models.These approaches differ in whether they localize task events, aggregate video-wide information, or learn temporal attention.
- Proposed Difference: The proposed approach learns task-agnostic skill-relevant temporal attention with a convolutional network and a rank-aware loss rather than recurrent networks.The paper cites concerns about preserving information over long videos and reports inferior recurrent-network performance in many sequence tasks.
- Attention Modules: The paper extends attention-based fine-grained recognition and video localization by training rank-specific attention instead of rank-agnostic attention.Existing temporal-attention work includes action recognition, localization, and class-specific attention.
- Architecture: The rank-aware network processes uniformly segmented ranked video pairs through I3D features, high- and low-skill attention branches, ranking functions, and uniform weighting.Its losses enforce pairwise ranking margins, improvement over uniform weighting, and rank-aware specialization.
3. Rank-Aware Attention Network
The network jointly learns ranking functions and temporal attention from video pairs, replacing assumptions of uniform or aligned segment importance with rank-aware weighting. Its two attention branches focus on high- and low-skill evidence, while multiple filters capture distinct skill-relevant subtasks.
- Problem formulation: Pairwise ranking supervision learns a function that ranks video p_i above p_j when p_i displays more skill.The training set contains annotated video pairs, enabling crowd-sourced ranking for generic tasks.
- Problem formulation: The model rejects uniform weighting and corresponding-segment comparisons because skill relevance, ordering, and performance can vary across videos.Some segments may show reversed rankings, while corresponding chunks may assume tasks occur in the same order and at the same speed.
- Rank-aware attention: Two attention modules learn separate scores for segments indicative of higher and lower skill, using rank-aware training to make their attentions diverge.The high-skill branch attends to higher-skill parts of the better video, while the low-skill branch attends to lower-skill parts of the other video.
- Overall network: The Siamese architecture processes video segments through high-skill, low-skill, and uniform-weighting branches, with separate ranking functions and combined ranking, disparity, and rank-aware losses.The disparity loss encourages an attention branch to distinguish skill better than uniform weighting and helps avoid attention local minima.
- Inference: During testing, a single video receives its rank from its ranking score, with the learned attention alone supplying the final representation rather than an additional uniform score.Shared weights are optimized across pairs in which the same video can appear at different relative skill levels.
- Multi-filter attention: Each attention module uses K filters whose segment weights are combined into a video representation, while diversity regularization encourages filters to attend to different informative subtasks.Without diversity regularization, filters may collapse onto the same discriminative segment; the loss also encourages sparse attention.
4. Tasks and Datasets
The evaluation uses EPIC-Skills and the larger BEST dataset to assess skill ranking across varied tasks, videos, environments, and viewpoints. BEST contains five tasks with 100 videos each and more annotated pairs than EPIC-Skills.
- EPIC-Skills covers surgery, doughrolling, drawing, and chopstick-using, with pairwise annotations for ranking videos within each task.
- BEST was collected from online videos to include more individuals, environments, viewpoints, and longer videos than the prior dataset.
- BEST contains five varied tasks—scrambling eggs, braiding hair, tying a tie, making an origami crane, and applying eyeliner—with 100 videos per task.
- Workers filtered YouTube videos for task relevance, quality, clear views, complete performances, and minimal edits before annotating skill suitability.
- Relevant activity boundaries were annotated with four-worker agreement, while pairwise skill orderings were retained only when all four workers agreed.
- The dataset uses 40% of possible pairings, balances video appearances across pairs, and exploits transitivity before adding comparisons between similarly ranked videos.
- BEST is considerably larger than EPIC-Skills in both videos and annotated pairs, with Table 1 also reporting average and standard-deviation video lengths.
5. Experiments
Experiments show that the proposed rank-aware attention model outperforms baselines across both datasets, with gains linked to its loss components, branch fusion, and selective temporal focus. The qualitative results further indicate that high- and low-skill modules attend to distinct skill-relevant or potentially irrelevant segments.
- Baselines and Attention: 4.3% and 5.4% gains over previous work are achieved on EPIC-Skills and BEST, respectively.
- Baselines and Attention: The final method outperforms every baseline on both EPIC-Skills and BEST.
- Ablation Study: The final rank-aware loss yields a 5% average improvement on BEST, including gains of 10.4% for Scramble Eggs and 8.8% for Apply Eyeliner.
- Branch Contribution: Fusing high- and low-skill branches improves performance by 2.9% on EPIC-Skills and 3.2% on BEST.
- Number of Filters: Most tasks peak at K = 3 filters, while additional filters add less-informative segments without further accuracy gains.
6. Conclusion
The paper presents rank-aware attention trained with a novel loss to identify informative video segments for skill determination. Across two datasets, including a newly introduced one, the method achieves state-of-the-art performance.
- The proposed model learns rank-aware temporal attention with a novel loss function.The loss identifies informative segments in relation to the skill shown in each video.
- The rank-aware loss learns which video segments are most informative for assessing skill.
- The disparity loss directly optimizes attention to select more informative segments than uniform weighting.It addresses instability when optimizing standard softmax attention for ranking.
- The method is evaluated on two datasets, including a dataset introduced by the paper.
- Average performance exceeds 80% on both datasets.