Source-linked AI summary
Groupwise Selective State-Space Filtering for Accurate and Streaming Action Boundary Detection
Mustafa Bora Çelik
TL;DR
Action boundary detection provides temporal units for untrimmed video without assigning action classes, but conventional segmentation combines localization with frame-level labels. This paper applies groupwise selective state-space scans to pre-extracted video features and reports improved boundary F1 over the strongest evaluated literature baseline on all three benchmarks, alongside a bounded-memory streaming variant.
Problem
Temporal action segmentation typically combines transition detection with assigning action classes, while boundary detection can instead provide unlabeled candidate intervals for downstream interpretation.
Source: §I · p. 1Method
The adapter processes pre-extracted video features with isolated groupwise selective scans, learned fusion, temporal modeling, and structured timestamp decoding.
Source: §I · p. 1Results
At the primary tolerance, the full pipeline improves over the strongest evaluated literature baseline by 0.087, 0.016, and 0.004 absolute F1 on Breakfast, GTEA, and 50Salads, respectively.
Source: B. Boundary-Detection Results · p. 3Takeaways & Limitations
The adapter localizes action transitions without semantic labels, and experiments establish boundary-localization performance on three benchmarks and measured feature-streaming operation.
Source: §IV · p. 4
Abstract
from arXiv · showhide
Action boundary detection partitions untrimmed video into intervals without assigning action classes. We present a boundary-detection adapter operating on pre-extracted video features, learning temporal representations via groupwise selective scans. Learned group fusion and temporal modeling convert these into transition scores, which are decoded into boundary timestamps. Trained with boundary-time supervision, the class-agnostic model is evaluated on Breakfast, GTEA, and 50Salads using temporal tolerances and bipartite matching, achieving boundary $F_1$ scores of 0.457, 0.622, and 0.611. A stateful variant enables feature-streaming inference with zero neural look-ahead, one-sample peak confirmation, and bounded memory. Downstream systems can subsequently assign s
I. INTRODUCTION
The adapter localizes action transitions in untrimmed video without assigning semantic labels, using boundary-time supervision and timestamp evaluation. Its full pipeline leads the completed baseline reproductions on mean primary boundary F1, while a stateful variant supports feature-streaming inference with bounded memory.
I. INTRODUCTION
- Boundary-time supervision trains the class-agnostic detector; candidate intervals may support downstream retrieval or video-language interpretation, a capability not evaluated here.The evaluation assesses boundary localization on Breakfast, GTEA, and 50Salads using their annotated transition times.Source: §I · p. 1
- Predicted and reference timestamps are matched one-to-one within fixed temporal tolerances, so boundary precision, recall, and F1 measure localization independently of action classes.Unmatched references count as false negatives and redundant predictions as false positives.Source: §I · p. 1
- The full pipeline achieves the best mean primary boundary F1 among completed baseline reproductions on Breakfast, GTEA, and 50Salads.The model is class-agnostic and localizes action transitions without assigning semantic labels to the resulting intervals.Source: §I · p. 1
- A stateful feature-streaming realization provides constant per-step computation and bounded O(1) recurrent memory, with zero neural look-ahead and no prefix recomputation.The streaming interface operates on arriving features; it does not establish raw-video causality.Source: §I · p. 1
A. From Interleaved Scanning to Temporal Group Chains
The method adapts the MamBOA scan principle to long I3D sequences by aligning corresponding feature subspaces into persistent, isolated temporal chains. Each chain carries its own recurrent state, preventing cross-group state sharing while preserving a trajectory across time.
A. From Interleaved Scanning to Temporal Group Chains
- The design adapts MamBOA [8] by replacing paired patch interleaving with persistent, mutually isolated channel-group chains for multitransition localization.This keeps corresponding subspaces adjacent along the temporal scan without cross-group leakage.Source: A. From Interleaved Scanning to… · p. 2
- The time-major feature layout is re-indexed into aligned temporal chains, so each recurrent state is updated only by the same subspace across successive times.The I3D descriptor has no explicit spatial grid, so each scanned unit is a channel group rather than an image patch.Source: A. From Interleaved Scanning to… · p. 2
- Each chain uses an independent forward Mamba block, with layer normalization within its 64-dimensional group and no recurrent state sharing.The selective state-space core includes a state-mediated term and additive input bypass; the implementation freezes D(g) = 0.Source: A. From Interleaved Scanning to… · p. 2
- Input-dependent transition and write terms regulate state retention and incorporation of the current group feature, producing trajectories rather than unrelated frame descriptors.Repeated patterns can preserve memory through small effective writes, while persistent changes rewrite the state.Source: A. From Interleaved Scanning to… · p. 2
B. Group Fusion and Boundary Formation
Groupwise scan outputs are fused into a compact boundary representation, contextualized, and converted to boundary scores. Soft boundary targets train the model, and structured decoding selects timestamps.
B. Group Fusion and Boundary Formation
- At each time step, restored group outputs are concatenated, and a learned projection maps the 2048-dimensional descriptor to a compact 64-dimensional boundary representation.Unlike mean or norm pooling, the projection can preserve which groups respond and learn their combination from boundary supervision.Source: B. Group Fusion and Boundary Formation · p. 2
- Three residual TCN layers use width-3 dilated convolutions and symmetric padding so each boundary score can use evidence on both sides of a candidate transition.A learnable 64→64→1 MLP maps the contextual representation to boundary logits.Source: B. Group Fusion and Boundary Formation · p. 2
- Reference boundaries supervise soft targets, and positive-weighted binary cross entropy trains the boundary logits.The soft targets accommodate slight timing deviations around annotations; σ is 0.8, 0.3, and 1.0 s for Breakfast, GTEA, and 50Salads, respectively.Source: B. Group Fusion and Boundary Formation · p. 2
- Penalized dynamic programming selects timestamps by maximizing cumulative standardized scores subject to a minimum separation and penalty.Backtracking, three-point parabolic sub-step refinement, and validation-calibrated temporal bias correction follow selection; streaming inference instead uses causal peak confirmation.Source: B. Group Fusion and Boundary Formation · p. 2
C. Stateful Feature Streaming
The streaming configuration removes the TCN and applies the boundary head directly to the groupwise representation. Persistent recurrent states and running normalization support causal feature-by-feature inference with bounded state.
C. Stateful Feature Streaming
- Strictly sequential operation uses a separate TCN-free configuration, applying the boundary head directly to z_t while retaining the groupwise Mamba formulation.Each arriving feature updates every group once; the model does not recompute the observed prefix or retain a growing feature cache.Source: C. Stateful Feature Streaming · p. 3
- The representation has zero neural look-ahead, but confirming a local peak requires one subsequent feature.Video-global score normalization is replaced by Welford running statistics, and the upstream I3D extractor's causality is outside the defined model.Source: C. Stateful Feature Streaming · p. 3
A. Protocol
The evaluation uses official dataset splits, common features, and timestamp-based one-to-one matching under dataset-specific tolerances. Results are summarized as video-macro F1, with variability reported across fold/seed runs.
A. Protocol
- Evaluation uses official splits for Breakfast, GTEA, and 50Salads, with identical 2048-dimensional features and dataset-specific subsampling.The official split counts are four folds for Breakfast and GTEA and five for 50Salads; subsampling factors are 1 and 2, respectively, for the named datasets.Source: A. Protocol · p. 3
- Predicted and reference boundaries are matched at most once using maximum-cardinality, minimum-error bipartite assignment, with dataset-specific narrow, primary, and wide tolerances.The primary tolerance is 1 s for Breakfast and 50Salads and 0.4 s for GTEA.Source: A. Protocol · p. 3
- Reported variability is the standard deviation across fold/seed runs, not an independent-sample confidence interval.The protocol reports video-macro F1.Source: A. Protocol · p. 3
- Table I reports primary action-boundary F1 under the common one-to-one protocol; Table II covers stateful streaming results on 50Salads.Table I marks best and second-best entries and uses N/C when a reproduction did not finish within the available compute budget; Table II specifies five folds, two seeds, zero neural look-ahead, and one-sample peak confirmation.Source: Table I · p. 3, Table II · p. 3
B. Boundary-Detection Results
The full pipeline achieves the best mean primary boundary F1 among the completed baseline reproductions on all three datasets. The 50Salads margin is small relative to run variability, so it is described as a best mean rather than an established significant advantage.
B. Boundary-Detection Results
- 0.087, 0.016, and 0.004 absolute F1 improvements over the strongest evaluated literature baseline on Breakfast, GTEA, and 50Salads, respectively, at the primary tolerance.The corresponding run standard deviations are 0.016, 0.030, and 0.046; the 50Salads margin is small relative to this variability.Source: B. Boundary-Detection Results · p. 3
- In the illustrated sequences, the adapter aligns closely with ground-truth transitions and suppresses ASRF's redundant predictions within uniform action intervals.Figure 2 compares representative sequences from GTEA, Breakfast, and 50Salads against ground truth and ASRF.Source: B. Boundary-Detection Results · p. 3
C. Streaming Results and Group Width
The streaming comparison varies group width, while the proposed width-64 configuration uses 32 temporal chains and supports feature-by-feature processing with bounded recurrent state. Figure 3 separately compares Breakfast boundary F1 with estimated complexity.
C. Streaming Results and Group Width
- Streaming processes one arriving feature at a time with zero neural look-ahead; peak confirmation occurs after one subsequent feature arrives.The width-64 implementation processes 50.5 steps/s with a p95 of 20.45 ms per feature, excluding the wait for peak confirmation and upstream I3D extraction.Source: C. Streaming Results and Group Width · pp. 3–4
- 45.7% F1 at approximately 1.756 GFLOPs on Breakfast, versus MS-TCN’s 34.9% at 1.740 GFLOPs and ASRF’s 37.0% at 2.810 GFLOPs.These are analytically estimated costs at reference sequence length T = 1092.Source: Figure 3 · p. 4
- Group width 64 yields the highest mean streaming score in the comparison with 0.658M parameters, though score differences are small relative to run variability.It forms 32 independent temporal chains; narrower projections use 20.3× fewer parameters than one full-descriptor block.Source: C. Streaming Results and Group Width · p. 4
D. Computational Complexity and Streaming Invariants
The paper estimates computational cost analytically and characterizes streaming cost and memory as bounded with respect to video duration. For fixed model dimensions, processing T features takes O(T) total computation.
D. Computational Complexity and Streaming Invariants
- At T = 1092 on Breakfast, the full pipeline uses approximately 1.756 GFLOPs and 0.70M parameters, with 10.8 percentage points higher F1 than MS-TCN at comparable estimated cost.The estimate is 1.60× fewer FLOPs than ASRF and more than 2× fewer than ASFormer.Source: D. Computational Complexity and… · p. 4
- The width-64 configuration is estimated at 1.534 MFLOPs per feature, or 1.675 GFLOPs for T = 1092.Counts are analytical estimates; the selective scan uses a 9TEN proxy, while several operations and I3D extraction are excluded.Source: D. Computational Complexity and… · p. 4
- For fixed model dimensions, each feature requires O(1) work and persistent working memory is O(1) with respect to video length; processing T features takes O(T) total computation.Persistent state includes 160 KiB of FP32 convolution/SSM state plus fixed-size normalization and decoder buffers.Source: D. Computational Complexity and… · p. 4
E. Component Ablation on 50Salads
The component study reports lower mean scores when normalization, learned fusion, or temporal context is removed, while the conclusion limits its claims to boundary localization and measured feature streaming. Downstream retrieval or VLM gains remain future work.
E. Component Ablation on 50Salads
- Table III covers five folds and two seeds; component variants were trained in a separate batch from the full-pipeline result.The aggregate scores are descriptive comparisons, not paired estimates of component effects.Source: Table III · p. 4, E. Component Ablation on 50Salads · p. 4
- The no-TCN configuration has the lowest mean score in the table (0.559); uniform group averaging scores 0.578, and a linear boundary head scores 0.603.Removing input normalization, learned fusion, or temporal context also yields lower reported mean scores; the comparisons are descriptive, not paired component-effect estimates.Source: E. Component Ablation on 50Salads · p. 4
IV. DISCUSSION AND CONCLUSION
- The adapter localizes transitions without semantic labels, and the experiments establish boundary-localization performance and measured feature-streaming operation—not raw-video causality or downstream VLM gains.Evaluating whether detected intervals improve retrieval or VLM reasoning is future work.Source: §IV · p. 4