Source-linked AI summary
FIS-OT: Feature-Induced Optimal Transport for Unsupervised Action Segmentation
Linxiang Peng, Xinyao Qin, Jinhan Li, Di Yang, Jiangtao Wang
TL;DR
Unsupervised action segmentation needs to recover action categories and boundaries without labels, while existing OT approaches overlook local information and can reinforce noisy pseudo-labels. FIS-OT couples pseudo-label-independent local feature learning with adaptive structural transport in a cyclic optimization loop, with experiments reporting competitive or state-of-the-art performance across three datasets.
Problem
Existing OT-based action segmentation methods use global constraints that overlook local information and rely heavily on self-generated pseudo-labels, creating confirmation bias from early noise.
Method
FIS-OT combines a triplet-loss Feature Enhanced Generator with a feature-induced residual structural prior and cyclic optimization linking local feature learning to global OT alignment.
Results
FIS-OT achieves competitive and state-of-the-art performance on Breakfast, 50Salads, and Desktop Assembly, with improvements in boundary accuracy and short-action segmentation.
Takeaways & Limitations
Integrating feature-driven structure into optimal transport provides a more adaptive and reliable approach to unsupervised video segmentation.
Abstract
from arXiv · showhide
Unsupervised action segmentation is a challenging task. It involves finding action categories and boundaries in videos without labels. Existing Optimal Transport (OT) methods use global constraints. This causes them to overlook the use of local information. Furthermore, existing Optimal transport architectures are prone to confirmation bias because they overly trust the pseudo-labels they generate. This causes models to learn from noise in the early training stages. To address these issues, we propose FIS-OT. It is a novel Feature-Induced Structured Optimal Transport framework. First, we introduce a Feature Enhanced Generator (FEG) module. It serves as an internal regularizer. By using triplet loss, FEG captures local consistency. It provides robust supervision that is independent of noisy pseudo-labels. Second, we propose a Feature-Induced Residual Structural Prior. This combines a fixed temporal backbone with dynamic feature similarities. This design ensures temporal continuity. It also allows the solver to adapt to complex action structures. Finally, we establish a cyclic optimization loop. This aligns local feature learning with global structural alignment. Extensive experiments on the three datasets show the effectiveness of our method.
I. INTRODUCTION
Unsupervised action segmentation supports applications such as instructional-video analysis, but OT methods can miss local temporal cues and reinforce inaccurate pseudo-labels. FIS-OT couples pseudo-label-independent local feature learning with adaptive global structural alignment in a cyclic process.
- Unsupervised action segmentation discovers action categories and boundaries in long, untrimmed videos without labels and supports video understanding applications.
- OT methods generate pseudo-labels for self-training, while ASOT formulates segmentation as fused Gromov–Wasserstein transport without presupposing action order.
- ASOT’s rigid structural prior smooths short actions and rapid transitions, biasing predictions toward long segments when frame features are noisy.
- ASOT’s reliance on its own inaccurate pseudo-labels can amplify errors because it lacks an independent supervision source.
- FIS-OT introduces a pseudo-label-independent Feature Enhanced Generator and a feature-induced residual structural prior combining a stable temporal backbone with feature affinity.
- The local and global components form a cyclic optimization process that reduces confirmation bias and improves segmentation of short and long actions.
II. RELATED WORK
Related work spans supervised temporal models and unsupervised video-level or activity-level methods. Unsupervised approaches use clustering, boundary detection, representation learning, and temporal regularization to discover actions without manual labels.
- Fully supervised methods provide reliable performance but require expensive frame-wise annotations, motivating unsupervised alternatives.
- Video-level approaches: Video-level unsupervised methods process videos individually using clustering or boundary detection based on visual similarities.
- Video-level approaches: TW-FINCH combines temporal proximity and semantic similarity for hierarchical clustering, whereas ABD detects boundaries from adjacent-frame similarities.
- Activity-level approaches: Activity-level methods learn representations self-supervisedly and then cluster embeddings, typically assuming a predefined number of clusters.
- Activity-level approaches: Activity-level research also uses temporal regularization, visual reconstruction, discriminative embedding losses, and self-supervised action-prototype discovery.
C. Optimal Transport for Action Segmentation
OT methods align video sequences and action distributions, but existing approaches can impose rigid temporal structure and rely heavily on self-generated labels. FIS-OT adds local feature cues and cyclic collaboration to make transport guidance more adaptive.
- TOT uses temporal OT to generate self-training pseudo-labels but assumes a fixed action ordering across videos, while UFSA still requires an estimated action order.
- ASOT avoids action-order assumptions through fused Gromov-Wasserstein OT, yet rigid global constraints overlook local cues and hinder short-duration action detection.
- FIS-OT introduces a Feature Enhancing Generator for intrinsic local cues independent of noisy pseudo-labels and a Feature-Induced Residual Structural mechanism for adaptive transport guidance.
- The FEG mines temporal and semantic frame similarities with triplet loss, while the OT decoder uses learned features to construct a structural cost matrix and solve a GW problem.
- Dual-level cyclic optimization lets local feature structure guide OT and global temporal constraints guide feature learning, breaking confirmation bias in self-training loops.
B. Temporal-Semantic Representation Learning
FIS-OT learns normalized frame representations with a Feature Enhanced Generator that models temporal–semantic similarity and regularizes local continuity without relying on noisy pseudo-labels.
- The FEG module maps frame features into a normalized latent space to prevent collapse and capture local action continuity.
- Hybrid Similarity Measure: Its target similarity distribution fuses temporal proximity with semantic similarity using Gaussian kernels.The temporal and semantic terms use bandwidths σt and σs, respectively.
- Triplet Selection & Loss: FEG selects top-K similar frames as positives and semi-hard samples as negatives for each anchor frame.
- Triplet Selection & Loss: The KL-divergence-based triplet loss pulls semantically consistent frames together and pushes dissimilar frames apart locally.This establishes a feature foundation for subsequent global segmentation.
C. Feature-Induced Residual Structural OT
FIS-OT formulates segmentation as FGW-based alignment between frames and action prototypes, using matching and structural costs with a residual prior that combines temporal structure and feature affinity.
- The method seeks an optimal transport plan assigning N video frames to K action prototypes.
- Its FGW objective combines matching cost, structural cost, entropy regularization, and a marginal-distribution penalty.
- Matching Cost (Ck): The matching cost measures cosine distance between frame embeddings and action prototypes.
- Residual Structural Prior (Cv): The residual structural prior integrates a rigid temporal skeleton with dynamic feature affinity instead of using only a fixed temporal band.The design targets nonlinear variations within actions while retaining temporal structure.
- A hard temporal window prevents structural collapse during early training when features are ambiguous.
1) Stability:
The feature-induced structure strengthens connections between temporally nearby, feature-similar frames while mutual exclusivity discourages assigning similar frames to different clusters.
- Stability: Feature affinity lets the OT solver represent internal action structure, including repetitive subactions.
- Stability: Within the temporal window, high feature similarity strengthens connections and promotes assignment to the same action cluster.
- Stability: Mutual exclusivity among action prototypes is imposed to prevent mode collapse.
- Stability: The mutual-exclusivity structure applies a heavy penalty when similar frames are assigned to different action clusters.
- Stability: The log-domain stabilized Sinkhorn algorithm solves the transport problem and produces soft pseudo-labels Q = T*.
D. Curriculum Optimization Strategy
FIS-OT uses a two-stage curriculum that first learns a coarse temporal skeleton, then activates feature guidance so FEG can refine the structural matrix and segmentation boundaries.
- Randomly initialized features can create a cold-start problem because the feature-induced structure depends on high-quality representations.
- Warm-up Phase: During warm-up, γ = 0 and strong fixed temporal priors guide learning of a coarse action skeleton.
- Refinement Phase: After warm-up, γ > 0 enables FEG features to refine the OT structural matrix and align boundaries with the video’s semantic manifold.
- The total objective combines cross-entropy between network predictions and OT pseudo-labels with FEG regularization.
IV. EXPERIMENTS
FIS-OT is evaluated on three unsupervised action-segmentation benchmarks using standard clustering-mapping and complementary frame, boundary, and class-balanced metrics. It achieves strong results, especially for short actions and boundary alignment, while remaining limited by noisy or occluded videos.
- Evaluation Metrics: MoF measures correctly predicted frames but can overlook short-segment errors because long action classes dominate the frame count.This motivates complementing MoF with boundary-sensitive and class-balanced metrics.
- Evaluation Metrics: Segmental F1 evaluates boundary quality, penalizes over-segmentation, and assesses detection of short actions.mIoU instead averages prediction–ground-truth overlap equally across action categories.
- Experimental Setting: FIS-OT is compared with clustering-based, temporal optimal-transport, and recent state-of-the-art unsupervised frameworks on Breakfast, 50Salads, and Desktop Assembly.Hungarian Matching maps predicted clusters to ground-truth labels at video or activity level.
- Limitations: Severe occlusions and local frame noise limit performance in the 50Salads (Mid) setting because OT remains sensitive to noise within individual videos.The paper states that FIS-OT improves upon ASOT but remains within the OT framework.
- Results: +6.7% F1 over ASOT on 50Salads (Mid) reflects improved short-action detection, while mIoU gains reach +4.1% on Breakfast and +5.2% on 50Salads (Eval).The reported short-action comparison is 39.0% versus 32.3%.
D. Ablation Study
Ablation results on Desktop Assembly show that FEG, the feature-induced structure, and curriculum warm-up each materially support FIS-OT’s segmentation quality. FEG has the largest measured contribution, while warm-up provides a stable starting structure.
- Ablation Protocol: The ablation study reports both Video-Level and Activity-Level metrics to assess local consistency and global semantic understanding.It is conducted on the Desktop Assembly dataset.
- Effect of Feature Enhanced Generator (FEG): Removing FEG causes the largest ablation loss, reducing Activity-Level F1 from 71.7% to 58.2%, a 13.5% drop.The authors associate this degradation with reliance on global OT alignment amid noisy video features.
- Impact of Feature-Induced Structure: Replacing the feature-induced structure with a fixed temporal mask decreases Activity-Level F1 by 6.3%.The dynamic feature-induced prior adapts segmentation boundaries to actions with different speeds and video content.
- Necessity of Warm-up Strategy: Skipping curriculum warm-up reduces Activity-Level F1 to 60.6%.The reported interpretation is that fixed priors first establish a coarse structural skeleton before feature-based refinement.
E. Qualitative Results And Analysis
Qualitative comparisons across Breakfast, 50Salads, and Desktop Assembly show FIS-OT producing less noisy and more fine-grained action timelines than ASOT. The clearest difference appears when short assembly steps are otherwise merged into one long segment.
- Breakfast: On Breakfast, FIS-OT reduces the segmentation noise observed in ASOT and produces more consistent predictions.This comparison is based on the qualitative timeline examples.
- 50Salads: On 50Salads, FIS-OT identifies actions at the beginning and end of the video more accurately than ASOT, which merges continuous steps into large blocks.The passage describes this as a qualitative improvement over ASOT.
- Qualitative Comparison: Figure 3 compares ground truth, FIS-OT predictions, and ASOT predictions across datasets and activities using color-coded action categories.The figure is intended for qualitative comparison rather than a reported numerical metric.
- Desktop Assembly: On Desktop Assembly, FIS-OT detects fine-grained short assembly steps and recovers their flow, whereas ASOT outputs a single long segment.The FIS-OT timeline aligns closely with the ground truth in the middle section.
- Overall Analysis: Across the experiments, coupling local representation learning with global structural alignment is reported to improve boundary accuracy and short-action segmentation.The conclusion characterizes the resulting solution as more adaptive and reliable for unsupervised video segmentation.