Source-linked AI summary
See the Change, Keep the Flow: Unsupervised Action Segmentation via Spectral-Temporal Representation Learning
Yun Li, Jun Xiao, Cong Zhang, Kin-Man Lam
TL;DR
OT-based unsupervised action segmentation depends on a representation space that is both sensitive to action changes and coherent over local temporal progressions. SpecT-OT combines spectral reparameterization and temporal affinity regularization within unbalanced OT, and experiments show complementary gains across benchmarks.
Problem
OT pseudo-label reliability depends on learning representations that distinguish action changes while preserving local temporal coherence.
Method
SpecT-OT combines unbalanced OT with SRP, which uses Fourier-basis projector reparameterization, and TAR, which regularizes pairwise frame affinities.
Results
Combining SRP and TAR improves all 15 ablation entries over the baseline by an average of 4.8 points and outperforms the stronger single-module variant on 14 entries.
Takeaways & Limitations
SpecT-OT jointly improves transition discrimination and local temporal coherence, yielding more reliable OT pseudo-labels and consistent gains under full and per-video matching.
Abstract
from arXiv · showhide
Unsupervised action segmentation aims to discover latent action categories and their temporal organization without action annotations. Optimal transport-based methods provide structured frame-to-action assignments, however, their pseudo-label quality is fundamentally conditioned on the representation space used to construct the transport cost. We argue that reliable OT pseudo-labeling requires a representation geometry that is simultaneously sensitive to discriminative action changes and coherent along local temporal progressions. Based on this insight, we propose SpecT-OT, a spectral-temporal representation learning framework built upon an unbalanced optimal transport pseudo-labeling concept. SpecT-OT introduces a Spectral Reparameterization Projector (SRP), which parameterizes projector weights with fixed Fourier bases and learnable coefficients to improve the modeling of rapidly varying discriminative features, and Temporal Affinity Regularization (TAR), which imposes distance-aware, label-free constraints on pairwise frame affinities to stabilize local temporal structure. The two components jointly produce more discriminative and temporally stable transport costs, yielding more reliable pseudo-labels for iterative representation learning. Experiments on four benchmarks demonstrate strong performance compared with state-of-the-art methods. SpecT-OT achieves the best results on 13 of 15 metrics, including 4.1-point MoF and 7.4-point F1 gains over the baseline on Breakfast and Desktop Assembly, respectively.
I. INTRODUCTION
Unsupervised action segmentation must discover frame-wise actions and temporal transitions without dense annotations. SpecT-OT improves OT pseudo-labeling by learning representations that capture discriminative action changes while preserving local temporal coherence.
- Unsupervised action segmentation discovers latent frame-wise action categories and temporal transitions without dense annotations.
- OT infers soft frame-to-action assignments whose optimized transport plan serves as pseudo-label supervision for representation learning.
- Relaxing OT marginal constraints accommodates unequal action durations and missing actions but does not ensure reliable assignments.
- Reliable pseudo-labeling requires representations sensitive to action changes and coherent across neighboring frames.
- SpecT-OT combines SRP and TAR to improve the representation space used for frame–action matching.SRP uses Fourier bases and learnable coefficients to model rapid discriminative feature variations, while TAR uses hard temporal masks and distance-aware soft weights to stabilize local frame representations.
- Extensive experiments on four benchmarks report consistent improvements in frame-level and segment-level metrics, supporting complementary benefits from SRP and TAR.
A. Unsupervised Action Segmentation
Unsupervised action segmentation couples representation learning with latent assignment, with OT providing structured frame-to-action matching. SpecT-OT focuses on the representation space, combining Fourier reparameterization with direct temporal-affinity regularization.
- Unsupervised approaches discover latent actions from visual and temporal structure, often coupling representation learning with iterative latent assignment.
- OT provides structured frame-to-action transport, while unbalanced OT accommodates unequal action durations and missing actions.
- SRP represents projector weights with fixed Fourier bases and learnable coefficients to broaden modeled feature frequencies.
- TAR regularizes pairwise affinities directly in embedding space, complementing structural constraints in the OT backbone.
- SpecT-OT targets representations that distinguish action changes while remaining stable across temporally coherent frames.
A. Preliminaries
SpecT-OT discovers frame-wise latent action assignments from pre-extracted features without frame-level labels or related annotations, assuming the number of latent classes is known. It projects frames into a latent space, matches them with learnable action prototypes, and uses unbalanced OT to generate soft assignments for representation learning.
- SpecT-OT trains without frame-level labels, transcripts, timestamps, or action-order annotations.
- The number of latent action classes K is assumed known for each activity.
- SRP maps pre-extracted frame features into a latent representation space with L frames and input dimension D_in.
- Learnable action prototypes represent latent action categories for frame-to-action matching.
- Unbalanced OT matches projected frames and prototypes to produce soft frame-to-action assignments used as pseudo-label supervision.
B. Spectral Reparameterization Projector
SRP reparameterizes projector weights with fixed Fourier bases and learnable coefficients to counter conventional MLPs’ low-frequency preference. This frequency-diverse parameterization supports modeling both rapid discriminative variations and slower contextual components.
- Conventional MLP projectors favor smoothly varying components, potentially weakening rapid variations that distinguish adjacent actions.
- SRP represents each projector-layer weight matrix over fixed Fourier bases instead of directly optimizing it.
- The basis matrix B^(n) contains γ fixed Fourier bases, while Θ^(n) contains their learnable combination coefficients.
- Combining P phases with two frequency banks yields γ = 2F · P fixed Fourier bases.
- The SRP architecture is controlled by hidden dimension d_hid and the number of Fourier-reparameterized hidden layers N_FR.
- SRP exposes the projector to a richer spectrum, retaining discriminative high-frequency cues alongside informative low-frequency context.
C. Temporal Affinity Regularization
TAR regularizes the embedding space with label-free, distance-aware temporal affinities. Its complementary local attraction and nonlocal separation preserve neighboring-frame coherence without uniformly smoothing genuine action changes.
- Uniform smoothing can erase genuine action changes, whereas no temporal constraint can amplify fluctuations into fragmented assignments.
- TAR defines pairwise temporal affinities from row-wise normalized frame embeddings.
- A hard temporal mask separates local from nonlocal frame pairs using a temporal-window size δ.
- A soft mask weights frame pairs by temporal distance, assigning different strengths to local and nonlocal constraints.
- Closer frames receive stronger coherence constraints, while increasingly distant frames receive stronger separation constraints.
- The local objective stabilizes neighboring affinities, while the nonlocal objective prevents all embeddings from collapsing into one representation.
- TAR produces clearer local affinity structures and reduces fragmented assignments without erasing meaningful action changes.
D. OT-Based Pseudo-Label Learning
SpecT-OT retains unbalanced OT as its assignment backbone while improving the representation geometry used to construct transport costs. SRP, TAR, and OT form a coupled self-training process that converts structured representations into soft pseudo-labels for learning.
- Unbalanced OT accommodates unequal action durations and actions absent from individual videos by softly constraining the action-side marginal.
- SpecT-OT improves the representation space for transport-cost construction rather than introducing a new OT solver.
- Projected frame embeddings and action prototypes are ℓ2-normalized before their frame–action transport cost is computed.
- The optimized transport plan T* is detached from gradient computation and used as soft pseudo-label supervision.
- SRP improves frame–prototype cost discrimination, TAR stabilizes local temporal geometry, and unbalanced OT converts this structure into pseudo-labels.
- The pseudo-labels supervise the projector and action prototypes in a coupled self-training process.
A. Experiment Settings
SpecT-OT is evaluated on four benchmarks against representative unsupervised action-segmentation methods under full and per-video Hungarian matching. It ranks first on 11 of 15 full-matching metrics, while performance is less competitive on YTI under per-video matching.
- Datasets and Protocol: The evaluation covers Breakfast, 50 Salads, YouTube Instructions, and Desktop Assembly using publicly released pre-extracted features.The benchmarks include different dataset sizes, activities, annotation granularities, and action vocabularies.
- Datasets and Protocol: SpecT-OT is compared with representative methods from 2019–2026, including reproduced and common-input adapted baselines.Earlier results are taken from open-source reports, while five recent methods are reproduced; MulSclTE uses an adapted common input-feature setting.
- Main Results: 11 of 15 metrics are won by SpecT-OT under full matching, including a 4.1-point Breakfast MoF gain over ASOT.It also improves Desktop Assembly F1/mIoU over ASOT by 7.4/5.1 points.
- Main Results: Under per-video matching, SpecT-OT achieves the best or tied-best result on five metrics and second-best on one.It is particularly strong on segmental F1 for FS-Mid, FS-Eval, and Desktop Assembly, and on Breakfast MoF.
- Main Results: YTI is less competitive under per-video matching, so dataset-level improvements do not uniformly transfer to video-specific cluster alignment.This comparison marks a scope boundary for the reported gains.
C. Qualitative Analysis
Qualitative results show that SpecT-OT preserves temporally coherent action regions while retaining major action transitions. Ablations indicate complementary benefits from SRP and TAR, with performance depending on architecture and regularization settings.
- Qualitative Segmentation: SpecT-OT avoids frequent label switching within long action segments while keeping most major action transitions distinguishable.The qualitative evidence indicates that local coherence does not collapse predictions into uniformly smoothed segments.
- Qualitative Segmentation: Short or visually similar operations remain difficult, including cutting and dressing actions on 50 Salads and the take-knife-to-cut-orange transition on Breakfast.Desktop Assembly also shows shifts or omissions for very short CPU-cover and fan operations.
- Ablation Results: Combining SRP and TAR improves all 15 ablation entries over the baseline by an average of 4.8 points.The combined model exceeds the stronger single-module variant on 14 entries; TAR alone is 0.1 point higher on FS-Eval mIoU.
- Ablation Results: SRP and TAR each improve 14 of 15 entries, with average gains of 2.8 and 3.3 points, respectively.These results support complementary transition-discriminative projection and local affinity regularization.
- Sensitivity and Embedding Analysis: The best SRP width is dataset dependent, one or two Fourier-reparameterized layers suffice for most datasets, and TAR is most reliable with w_TAR between 10^-4 and 10^-3.A large TAR weight can suppress useful discriminative variations, while an excessively small weight weakens regularization.
- Sensitivity and Embedding Analysis: SRP produces clearer block-diagonal frame affinity while preserving within-segment similarity.The visualization compares frame-wise cosine-distance matrices with and without SRP; darker colors indicate greater embedding similarity.
E. Efficiency Analysis
SpecT-OT remains within a practical single-GPU memory budget, with limited measured runtime overhead despite higher nominal GFLOPs. Its spectral-temporal components support consistent benchmark improvements across multiple evaluation settings.
- Resource Usage: 5.47–7.76 GB of memory keeps SpecT-OT within a practical single-GPU budget on an RTX 4090.Compared with ASOT, the measured per-epoch runtime changes only slightly despite higher nominal GFLOPs.
- Runtime: SRP’s parallel matrix operations do not translate proportionally into wall-clock overhead, with SpecT-OT even faster on Breakfast and YTI.
- Design: SpecT-OT combines Fourier-based SRP with label-free, distance-aware TAR to model discriminative changes and preserve local temporal coherence.Together, the components improve the representation space used for OT frame–action matching.
- Empirical Results: Best results on 11 of 15 metrics across four public benchmarks demonstrate complementary benefits from SRP and TAR.The improvements hold under both full and per-video matching.