PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Action Segmentation: An Analysis of Modern Techniques

Guodong Ding, Fadime Sener, Angela Yao

arXiv:2210.10352v5cs.CV

TL;DR

Temporal action segmentation lacks a systematic survey despite rapidly expanding methods for densely labeling long, multi-action videos. This paper synthesizes TAS definitions, datasets, supervision, modeling techniques, evaluation, and research gaps, finding that current datasets often underrepresent repetition and ordering variation while posing long-tailed challenges.

  • Problem

    TAS has rapidly expanded without a systematic survey, while existing datasets often provide limited action repetition and order variation and exhibit long-tailed distributions.

  • Method

    The survey organizes TAS by task foundations, datasets, supervision levels, frame representation, temporal modeling, evaluation measures, and existing methods.

  • Results

    The analysis finds that TAS datasets are often limited in action repetition and order variation, with long-tailed action distributions creating extra challenges.

  • Takeaways & Limitations

    The survey provides standardized evaluation guidance for unsupervised segmentation and introduces dataset-dynamics metrics for repetition and order variation.

  • Takeaways & Limitations

    MoF can obscure class imbalance and fragmented segments, so class-balanced and segment-based measures may be more appropriate.

Abstract

from arXiv · show

Temporal action segmentation (TAS) in videos aims at densely identifying video frames in minutes-long videos with multiple action classes. As a long-range video understanding task, researchers have developed an extended collection of methods and examined their performance using various benchmarks. Despite the rapid growth of TAS techniques in recent years, no systematic survey has been conducted in these sectors. This survey analyzes and summarizes the most significant contributions and trends. In particular, we first examine the task definition, common benchmarks, types of supervision, and prevalent evaluation measures. In addition, we systematically investigate two essential techniques of this topic, i.e., frame representation and temporal modeling, which have been studied extensively in the literature. We then conduct a thorough review of existing TAS works categorized by their levels of supervision and conclude our survey by identifying and emphasizing several research gaps. In addition, we have curated a list of TAS resources, which is available at https://github.com/nus-cvml/awesome-temporal-action-segmentation.

1 INTRODUCTION

Temporal action segmentation (TAS) assigns frame-wise action labels to temporally untrimmed, often minutes-long videos, addressing limitations of short-clip action recognition. This survey organizes TAS methods, analyzes their representations and temporal modeling, and identifies dataset and evaluation gaps.

  • Motivation: TAS segments temporally untrimmed videos into successive actions with frame-wise labels, extending video understanding beyond pre-trimmed clips.The task supports understanding action timing, progression, and transitions in settings such as surveillance and human-robot interaction.
  • Motivation: Unlike short-clip action recognition, TAS handles multiple related actions that may span several minutes in real-world video streams.Standard action-recognition approaches are not directly applicable because individual actions are temporally extended and interrelated.
  • Technical challenges: Effective TAS requires discriminative frame representations and temporal models that capture action boundaries, ordering, repetition, and duration.The survey frames representation learning and sequential modeling as two central technical considerations.
  • Survey scope: This work presents the first survey focused specifically on TAS and proposes a taxonomy emphasizing the contributions of existing methods.It distinguishes TAS research from surveys centered on action recognition, localization, or anticipation.
  • Survey contributions: The survey introduces repetition and order variation scores, distinguishes evaluation settings, and provides standardized unsupervised and class-based evaluation proposals.It also identifies that many existing datasets have limited action repetition and order variation.

2 A PRIMER ON SEGMENTATION

This section establishes the survey’s technical scope: TAS foundations, datasets, supervision regimes, and evaluation metrics.

  • Section scope: The primer covers technical foundations, popular training and evaluation datasets, supervision levels, and performance metrics for TAS.These topics provide the basis for comparing segmentation methods and their empirical results.

2.1 Definition

TAS partitions an untrimmed video into temporally ordered segments with predefined action labels, equivalently representing the result as frame-wise classifications. It differs from related tasks through its combination of dense labels, boundary localization, semantic understanding, and temporal reasoning.

  • Definition: Given a video of length T, TAS outputs consecutive segments, each represented by an action category and segment length.The segment formulation assigns each segment s_n a label c_n and length ℓ_n.
  • Task distinctions: TAS uniquely combines temporal relations, action-boundary localization, semantic segment understanding, and dense frame-wise labeling.The task landscape distinguishes TAS from neighboring video and sequence-segmentation problems along these dimensions.
  • Definition: TAS can equivalently be formulated as frame-wise action classification, with each frame assigned one predefined action label.Segment-level and frame-wise formulations can reconstruct one another.
  • Supervision: The segment formulation is common in weakly supervised methods, whereas fully supervised methods commonly use frame-wise labels.The distinction follows the availability of dense annotations.
  • Related tasks: Temporal action localization detects action instances from sparse annotations, while TAS produces dense frame-wise labels for untrimmed videos.Related tasks differ in annotation density and output granularity.
  • Related tasks: Sequence segmentation, key-frame detection, complex activity classification, and generic event-boundary detection differ from TAS in data dimensionality, boundary goals, semantic labels, or temporal assumptions.These neighboring tasks respectively segment lower-dimensional sequences, identify characteristic frames, classify whole activities, or detect class-agnostic boundaries.

2.2 Datasets

The survey compares procedural video datasets by annotation, scale, viewpoint, temporal dynamics, and distributional challenges. It finds limited domain diversity, substantial class imbalance, and varying action repetition and ordering across benchmarks.

  • Dataset Overview: Core TAS datasets annotate action segments with start times, end times, and class labels, primarily covering procedural activities such as meal preparation and furniture assembly.
  • Dataset Characteristics: The benchmarks differ in recording setup: GTEA uses a cap-mounted camera, 50Salads a top-down view, Breakfast multiple third-person cameras, and Assembly101 provides synchronized egocentric and third-person views.
  • Related Datasets: Several related datasets have limited suitability for TAS because they lack dense frame-wise labels or use annotations incompatible with high-level action-dynamics modeling.Epic-Kitchens lacks frame-wise dense action labels, while other datasets may provide only fine-grained movement labels or temporal boundaries and textual descriptions.
  • Dataset Limitations: Online video collections offer scale and diversity but may contain editing artifacts and domain gaps that limit applicability to real-time action anticipation or early detection.
  • Dataset Limitations: Recorded procedural datasets remain domain-limited: most focus on cooking and kitchen activities, with Assembly101 identified as the only large-scale dataset extending beyond cooking.
  • Background Frames: Task-irrelevant background frames occur at arbitrary locations and varying lengths in several YouTube datasets and are commonly treated as an ordinary class during training and inference.
  • Long-Tailed Distributions: Action durations and frequencies are highly imbalanced: Breakfast’s ‘fry pancake’ is 639× more frequent than ‘take butter’, while Assembly101 has an imbalance ratio of 2604.
  • Temporal Dynamics: The survey characterizes temporal dynamics using repetition and order variation scores; Assembly101 has relatively more repeated steps and an intermediate order variation compared with Breakfast and 50Salads.Higher repetition scores indicate more action repetition, while lower order variation scores indicate looser ordering constraints.

2.3 Supervision

TAS spans supervision regimes from dense frame labels to increasingly sparse weak labels and activity-grouped unsupervised training. These regimes trade annotation effort against the amount and structure of label information available.

  • Supervision regimes: Fully supervised TAS uses dense action labels for every frame, requiring annotators to view the entire training video sequence.Semi-supervised TAS reduces effort by densely labeling only some videos and treating the rest as unlabeled.
  • Supervision regimes: Weak supervision progressively removes information, moving from timestamp exemplars to ordered action transcripts and then unordered action sets.Each step requires less annotation effort than dense video labels.
  • Supervision regimes: Unsupervised TAS groups videos depicting the same activity, so it still uses activity labels even though frame-level action labels are absent.Its distinction from weak activity-label supervision lies in how video collections are processed during training.

2.4 Evaluation

TAS evaluation combines frame-wise accuracy with segment-sequence and overlap measures, while unsupervised evaluation additionally requires matching clusters to semantic labels. Matching scope changes the learning objective from intra-video discrimination to broader activity-wide and global correspondences.

  • Evaluation metrics: MoF measures correct frame predictions, whereas Edit and F1 scores evaluate segment sequences or segment overlap.All three are used in fully, weakly, and semi-supervised settings; unsupervised work reports F1 and MoF.
  • Evaluation metrics: MoF can obscure qualitative differences under long-tailed class imbalance and does not capture segment quality.A model may achieve high MoF despite poor segmentation of individual segments.
  • Evaluation metrics: The Edit score uses normalized Levenshtein distance to assess action-sequence similarity without requiring exact frame-wise correspondence.Dynamic programming computes the minimum insertions, deletions, and replacements needed to transform one segment sequence into another.
  • Unsupervised matching: Unsupervised evaluation uses Hungarian matching to associate cluster frames with semantic action labels before computing TAS metrics.When cluster and label counts differ, unmatched clusters are treated as background automatically.
  • Matching scopes: Video-level matching produces the highest scores because it only requires action associations within one video, whereas activity-level matching shares associations across videos in an activity.Activity-level grouping can therefore change label assignments across videos.
  • Matching scopes: Global-level matching is most challenging because it must establish correspondences both within and across complex activities.Broader scopes require increasingly general action associations and learning objectives.

3 CORE TECHNIQUES

Modern TAS systems rely on frame-wise representations combined with temporal and sequential modeling. The survey covers feature extraction, recurrent and convolutional temporal models, transformers, probabilistic sequence models, and boundary refinement.

  • Core techniques: TAS uses frame-wise representations for spatial appearance and motion, then temporal or sequential modeling to incorporate dependencies and context.Pre-computed features remain standard because they reduce computational demands and isolate architectural comparisons.
  • Frame representation: Handcrafted trajectories and deep backbones such as I3D provide alternative frame-level feature representations for video understanding.IDT captures trajectory statistics with Fisher Vectors, while I3D inflates 2D kernels into spatiotemporal kernels.
  • Temporal modeling: Temporal modeling expands receptive fields and exchanges information across frame-wise features using RNNs, TCNs, or transformers.RNN memory may be insufficient for long sequential relationships, while TCNs use pooling or dilated convolutions to enlarge temporal context.
  • Temporal modeling: ASFormer replaces ED-TCN convolutions with transformer blocks, using encoder self-attention and decoder cross-attention on pre-computed I3D features.Related transformer models can instead map frame inputs directly to action-sequence outputs.
  • Temporal modeling: Transformer use in TAS remains limited because models lack inductive biases, require large video corpora, and may learn weak attention over long input spans.Existing TAS datasets are relatively small for training large transformers effectively.
  • Sequential modeling: Segment-level sequence models capture procedural ordering through models such as HMMs and Mallows Models, while HMM formulations combine context, length, and visual components.Explicit action-length modeling helps prevent unreasonably long predicted segments.
  • Boundary refinement: Boundary refinement exploits local continuity and soft transitions to improve action segmentation near boundaries.Soft-boundary variants include linear and sigmoidal transitions.

4 LEVEL OF SUPERVISION

The survey organizes modern TAS methods by supervision level: fully supervised, weakly supervised, unsupervised, and semi-supervised approaches.

  • Levels of supervision: TAS techniques are categorized into fully supervised, weakly supervised, unsupervised, and semi-supervised approaches.The survey devotes a separate subsection to each supervision regime.

4.1 Fully-Supervised Approaches

Fully supervised TAS methods assume frame-level labels and advance along three axes: frame representation, temporal architecture, and iterative or boundary-focused refinement. The surveyed architectures evolve from deep feature-plus-temporal-model pipelines toward multi-stage TCNs and transformer-based designs.

  • Supervision: Full supervision labels every frame in every training sequence, supporting dense action segmentation.The section contrasts this setting with weaker supervision regimes.
  • Representation learning: Initial systems combine deep visual features with temporal models such as semi-Markov models or bidirectional LSTMs.ST-CNN uses CNN features with a semi-Markov model, while Bi-LSTM uses multistream appearance and motion features.
  • Representation learning: Representation-focused methods improve fine-grained action modeling through deformable convolutions, multimodal generative models, temporal aggregation, and prompt-based learning.These approaches target motion, auxiliary modalities, long-range observations, or video-text feature learning.
  • Temporal modeling: TCNs directly model durations, pairwise transitions, and long-term dependencies, with ED-TCN and TricorNet combining convolutional and recurrent components.ED-TCN uses temporal convolutional and deconvolutional kernels, whereas TricorNet replaces its decoder with a bidirectional LSTM.
  • Evaluation: Supervised TAS methods are compared on GTEA and Breakfast across representation, TCN, transformer, and refinement categories.The comparison is organized by the architectural or methodological focus of each method.
  • Temporal modeling: MS-TCN preserves full temporal resolution and iteratively refines predictions across stages, addressing detail loss from downsampled inputs.Subsequent methods modify this backbone with feature fusion, gated correction, sequence-level losses, coarse-to-fine decoding, attention, and boundary refinement.

4.2 Weakly-Supervised Approaches

Weakly supervised TAS reduces reliance on dense frame labels through transcripts, action sets, timestamps, or text. The surveyed methods generate or align pseudo frame labels, while comparisons indicate that stronger supervision generally produces better performance.

  • Supervision types: Weak supervision uses transcripts, action sets, timestamps, or text instead of dense frame-wise annotations.Transcripts and action sets list actions, timestamps label selected frames, and text includes narrations or related data.
  • Comparative performance: Timestamps-based approaches outperform the other compared weakly supervised approaches, while transcripts outperform action sets.The comparison covers Breakfast and 50Salads and reports a general association between supervision level and performance.
  • Transcripts: Transcript methods either iteratively refine initial frame labels or align transcripts with frames in a single stage.Single-stage methods address initialization sensitivity and convergence concerns associated with iterative solutions.
  • Transcripts: Transcript-based methods use Viterbi decoding, discriminative alignment, segmentation graphs, or prototype-based losses to construct valid frame-level supervision.NN-Viterbi, D3TW, CDFL, and DP-DTW differ in how they model transcripts, candidates, energies, or action prototypes.
  • Action sets: Action-set methods infer possible ordered action sequences or directly predict segments, but candidate transcript generation may omit valid sequences and limit accuracy.SCT instead learns from sets with set prediction and consistency constraints.
  • Timestamps: Timestamp methods propagate sparse labels or infer missing labels, but several approaches assume every action instance receives a timestamp.RAS-TSS relaxes this assumption by allowing missing annotations and expanding predicted boundaries around timestamps.
  • Text supervision: Text-assisted methods combine visual and narrative vocabularies in generative models to detect actions shared across videos.The approach represents frames with binary histograms of visual and textual words.

4.3 Unsupervised Approaches

Unsupervised TAS operates without action labels but often relies on activity-specific collections or activity labels. Methods alternate representation learning and clustering, jointly learn embeddings and clusters, or detect boundaries without modeling action dynamics.

  • Setting: Unsupervised TAS uses no labels, although many methods operate on videos from one activity or require activity labels.The surveyed strategies include iterative representation learning and clustering or simultaneous learning procedures.
  • Two-Stage Learning: Two-stage methods alternate discriminative appearance learning with generative sequence modeling, with models differing in their treatment of order variation and repetitions.The gMM handles permutations, missing steps, and ordering variation but not repeated actions, whereas Prism handles repetitions while assuming a shared ordering.
  • Representation learning and clustering: Embedding-and-clustering methods learn temporal or visual-temporal representations before clustering frames and recovering action order, but fixed-order assumptions limit deviations and repetitions.Later methods use co-occurrence relations, self-supervised feature learning, or other embedding strategies to improve flexibility.
  • Joint Representation Learning and Clustering: Joint methods learn embeddings and clusters together, using positional encodings, contrastive learning, temporal optimal transport, coherence losses, or prototype alignment.UFSA combines segment-level prediction and frame-to-segment alignment to allow more flexible action ordering.
  • Boundary detection: Boundary-only methods detect temporal changes without modeling action dynamics and can outperform unsupervised temporal segmentation techniques on reported evaluations.The survey attributes this partly to small datasets or activities with predominantly fixed ordering.
  • Evaluation: Unsupervised methods are evaluated on Breakfast with comparisons spanning learning strategy, evaluation level, corpus scope, and flexibility to deviations, missing steps, and repetitions.The table distinguishes two-stage and joint learning, video-level and activity-level evaluation, and global-corpus methods.
  • Boundary detection: Self-supervised boundary methods use next-frame prediction discrepancies or temporally weighted hierarchical clustering, while CAD learns representations from latent action prototypes across the corpus.TW-FINCH operates directly on pre-computed features without training.

4.4 Semi-Supervised Approaches

Semi-supervised TAS uses dense frame labels for only a subset of videos and exploits the remaining unlabeled data for representation learning or prior mining. Reported methods use action-level consistency or pseudo-labeling to transfer information to unlabeled videos.

  • Setting: Semi-supervised learning provides dense frame-wise labels for a subset of videos and uses the unlabeled set for representations or priors.This supervision setting differs from weak supervision, which requires annotations for every training video.
  • Semi-supervised objectives: SemiTAS uses action affinity and continuity losses to transfer action composition, distribution, and temporal information between labeled and unlabeled videos.The affinity loss minimizes KL divergence between the closest labeled and unlabeled video pairs.
  • Semi-supervised objectives: ICC achieves comparable performance to fully supervised counterparts with only 40% of labeled videos by learning contrastive features and using predictions as pseudo-labels.The classifier is trained on self-supervised representations, and its predictions supervise the unlabeled videos.

5 CONCLUSIONS AND OUTLOOK

The survey identifies persistent limitations in TAS data, supervision, feature inputs, and modeling, while outlining downstream and online directions. It calls for segment-level learning and clearer action-boundary definitions to address oversegmentation and annotation uncertainty.

  • The survey provides a taxonomy, systematic methodology analysis, supervision-based work classification, and research-gap agenda for a rapidly evolving field.
  • Pre-computed visual features reduce computation but may favor stereotypical actions and limit representation diversity.
  • Most sequential models operate iteratively after frame-wise prediction; segment-level losses remain under-explored despite their potential to reduce oversegmentation.
  • Timestamp supervision can approach fully supervised performance, but action-boundary uncertainty still affects model performance and labeling practice.
  • TAS supports alerts, surgery-time estimation, action anticipation, and video summarization through semantically labeled temporal segments.
  • Online TAS remains limited by reliance on frame-wise pre-computed features, motivating end-to-end online segmentation.
Loading 2210.10352v5…