PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Action Segmentation

Daochang Liu, Qiyue Li, AnhDung Dinh, Tingting Jiang, Mubarak Shah, Chang Xu

arXiv:2303.17959v2cs.CVeess.IV

TL;DR

Temporal action segmentation labels every frame in an untrimmed video, and prior work often refines predictions through multi-stage models. DiffAct instead generates action sequences through conditional iterative denoising, and experiments report better-than or on-par results against state-of-the-art methods on three benchmarks.

  • Problem

    Temporal action segmentation must assign an action label to each frame of an untrimmed video, while prior approaches commonly use multi-stage iterative refinement.

    Source: §1 · p. 1

  • Method

    DiffAct generates framewise action-label sequences by iteratively denoising from random noise conditioned on video features, using condition masking to model position, boundary, and relation priors.

    Source: §1 · pp. 1–2

  • Results

    On GTEA, 50Salads, and Breakfast, DiffAct performs better than or on par with state-of-the-art methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that iterative denoising with condition masking provides a benchmark-performing framework for temporal action segmentation.

    Source: §1 · p. 2, §7 · p. 8

    DiffAct’s advantage on the small-scale GTEA dataset is less significant than on larger datasets, and the authors speculate that few videos make generative learning harder.

    Source: §6 · p. 8

Abstract

from arXiv · show

Temporal action segmentation is crucial for understanding long-form videos. Previous works on this task commonly adopt an iterative refinement paradigm by using multi-stage models. We propose a novel framework via denoising diffusion models, which nonetheless shares the same inherent spirit of such iterative refinement. In this framework, action predictions are iteratively generated from random noise with input video features as conditions. To enhance the modeling of three striking characteristics of human actions, including the position prior, the boundary ambiguity, and the relational dependency, we devise a unified masking strategy for the conditioning inputs in our framework. Extensive experiments on three benchmark datasets, i.e., GTEA, 50Salads, and Breakfast, are performed and the proposed method achieves superior or comparable results to state-of-the-art methods, showing the effectiveness of a generative approach for action segmentation.

1. Introduction

DiffAct reframes temporal action segmentation as conditional generation and uses iterative denoising with condition masking to model action priors. Experiments on three datasets show results better than or on par with state-of-the-art methods.

1. Introduction

  • Temporal action segmentation assigns an action label to every frame of an untrimmed video; prior multi-stage methods iteratively refine predictions, helping capture temporal dynamics and reduce over-segmentation errors [15].Examples include MS-TCN [20] [41] and ASFormer [73].Source: §1 · p. 1
  • DiffAct formulates framewise action labels as a conditional generation problem and refines predictions through iterative denoising, sharing the iterative-refinement paradigm of multi-stage models.The figure contrasts multi-stage prediction refinement with action-sequence generation through denoising.Source: §1 · p. 1, Figure 1 · p. 1
  • Condition masking is designed to jointly model position, boundary, and relation priors, including characteristic action timing, ambiguous transitions, and temporal ordering.During training, masking yields fully conditional, partially conditional, and unconditional learning conditions.Source: §1 · p. 2
  • On GTEA, 50Salads, and Breakfast, DiffAct performs better than or on par with state-of-the-art methods.The paper presents these experiments as evidence for diffusion-based temporal action segmentation.Source: §1 · p. 2

2. Related Works

  • Prior work addressed boundary ambiguity and relational structure with separate methods, while position priors were less explored; this paper uses condition masking to address all three together.Examples include boundary-aware pooling, boundary regression [31], graph or sequence-to-sequence relational reasoning [6], and annotation-derived constraints.Source: §2 · p. 2
  • A related GAN method synthesized intermediate framewise action codes [22], whereas DiffAct generates action-label sequences with diffusion; the paper presents diffusion as new to action analysis.Source: §2 · p. 3

3. Preliminaries

Diffusion corrupts data through a forward noise process and learns a reverse process that denoises it into generated samples. The preliminaries also describe conditional generation and sampling variants.

3. Preliminaries

  • The forward process progressively adds Gaussian noise to data according to a variance schedule, with a closed-form expression for obtaining a noisy sample directly from the original.The noise intensity depends on the cumulative schedule, and the noise term is sampled from a normal distribution.Source: §3 · p. 3
  • The reverse process starts from noise and removes it step by step; the paper’s model predicts the original data x0 rather than the reverse-step mean.Training uses a mean squared error between the predicted and original x0, with the diffusion step selected randomly at each iteration.Source: §3 · p. 3
  • At inference, a trained model iteratively updates a pure-noise sample toward x0; sampling strategies may skip steps in the trajectory for efficiency.Source: §3 · p. 3
  • Conditional generation adds conditioning information as an input to the diffusion model, with class labels, text prompts, and image guidance given as examples.Source: §3 · p. 3

4. Method

DiffAct formulates frame-wise action labeling as conditional generation and uses diffusion denoising to produce predictions. Its training-time condition masking targets position, boundary, and relation priors.

4. Method

  • DiffAct treats action segmentation as conditional generation, using video features to condition predictions of the frame-wise action sequence.An encoder enriches the input features with long-range temporal information and makes them task-oriented.Source: §4 · p. 3

4.1. Diffusion Action Segmentation

  • During training, the model learns to denoise action sequences corrupted with noise at a randomly selected diffusion step, using encoded video features as conditions.The denoised output gives action probabilities for each frame, and the diffusion step lets the decoder handle different noise intensities.Source: §4.1 · pp. 3–4
  • The training objective combines cross-entropy, temporal smoothness, and boundary-alignment losses to match denoised predictions with ground truth.Boundary alignment compares smoothed ground-truth transition probabilities with probabilities derived from adjacent-frame action predictions.Source: §4.1 · p. 4
  • At inference, the decoder iteratively denoises a random-noise sequence into the final action prediction, with encoded video features unmasked.A sampling trajectory that skips steps is used to speed inference.Source: §4.1 · pp. 4–5

4.2. Action Prior Modeling

  • Training randomly masks conditioning features to model action priors: position masking removes all visual features, boundary masking removes features near transitions, and relation masking hides segments of a random action class.The masks respectively encourage reliance on video length and frame positions, context around boundaries, and surrounding actions to infer missing segments.Source: §4.2 · p. 5

5. Experiments

Experiments compare DiffAct with state-of-the-art methods on three datasets, then examine its design choices, denoising behavior, and computational cost. Results show stronger average performance on 50Salads and Breakfast, comparable overall performance on GTEA, and gains across several ablations.

5.1. Setup

  • Experiments use GTEA, 50Salads, and Breakfast, with five-fold cross-validation on 50Salads and four-fold cross-validation on GTEA and Breakfast.The evaluation reports frame-wise accuracy, edit score, and F1 at 10%, 25%, and 50% overlap.Source: §5.1 · p. 5

5.2. Comparison to State-of-the-Art

  • Average performance improved from 86.2 to 87.4 on 50Salads and from 73.9 to 75.1 on Breakfast; GTEA results were comparable overall.On GTEA, DiffAct had higher accuracy and F1@50.Source: Table 1 · p. 6, §5.2 · p. 6

5.3. Ablation Studies

  • Using all three modeled priors gives the best performance, with the position prior especially useful.The ablation varies masking schemes for the position, boundary, and relation priors.Source: §5.3 · p. 6, Table 2 · p. 7
  • All loss components are needed for the best result, and the boundary alignment loss improves frame-wise accuracy and temporal continuity beyond Lce and Lsmo.The compared variants include the full Lsum, Lsum without Lbd, Lsum without Lsmo, and vanilla Lce.Source: §5.3 · p. 6, Table 3 · p. 7
  • Performance rises with more inference steps but with diminishing marginal benefits, while computation grows linearly; the method uses 25 steps for a performance-cost balance.Table 4 reports results for different inference-step counts.Source: §5.3 · pp. 6–7, Table 4 · p. 7
  • Using input video features directly as the condition drops performance, encoder-layer choice has little effect, and DiffAct remains superior with an MS-TCN backbone.The MS-TCN variant uses a single-stage TCN encoder and a step-embedded decoder.Source: §5.3 · p. 7, Table 6 · p. 7

5.4. Qualitative Result and Computational Cost

  • Stepwise predictions refine random noise and gradually correct a broken “cut cucumber” segment; predictions remain reasonably stable across tested inference masks.The mask comparison is for analysis only because boundary and relation masks depend on ground truth.Source: §5.4 · p. 7, Figure 4 · p. 7, Table 8 · p. 8
  • With 8 inference steps, the lightweight decoder uses fewer FLOPs than ASFormer; 25 steps further improves results at an acceptable overhead.Table 7 compares computational costs and average performance on 50Salads.Source: §5.4 · p. 7, Table 7 · p. 8

6. Discussion

Masked-condition analyses indicate that DiffAct models position, boundary, and relation priors, while its advantage is less pronounced on small-scale GTEA than on larger datasets.

6. Discussion

  • 34.8 edit score under the completely masked condition M_P exceeds random guess, indicating learned correlations between actions and time locations.With M_B or M_R applied, the model still achieves reasonably good performance, suggesting boundary and relation priors are handled well.Source: §6 · p. 8
  • With M_P, predictions remain generally plausible; M_B preserves action-boundary detection, and M_R restores the missing “cut tomato” action.The examples include cutting and placing ingredients in the video middle and mixing and serving at the end.Source: §6 · p. 8
  • DiffAct’s advantage is less significant on small-scale GTEA than on large datasets, where it makes clear edit-score gains.The authors speculate that few videos make generative learning of action-sequence distributions more difficult and suggest model reassembly or action-statistics-based perturbations as possible remedies.Source: §6 · p. 8

7. Conclusion

The conclusion presents a temporal action-segmentation framework that generates action sequences through iterative denoising. It combines masking to exploit three human-action priors and reports state-of-the-art benchmark results.

7. Conclusion

  • The framework generates action sequences through an iterative denoising process.The conclusion describes this as a new framework for temporal action segmentation.Source: §7 · p. 8
  • A flexible condition-masking strategy is designed to jointly exploit position, boundary, and relation priors of human actions.Source: §7 · p. 8
  • The framework achieves state-of-the-art results on benchmark datasets.The conclusion links this outcome to iterative refinement, generative modeling, and enhancement of the three action priors.Source: §7 · p. 8

Diffusion Action Segmentation: Supplementary

The supplementary material collects implementation details, experimental comparisons, qualitative results, and other early attempts the authors consider interesting.

Diffusion Action Segmentation: Supplementary

8. More Implementation Details

Implementation details specify dataset-specific Gaussian smoothing for soft boundary ground truth, decoder configurations, the absence of positional encoding, and the training hardware.

8. More Implementation Details

  • Gaussian-kernel standard deviations for soft boundary ground truth are 1, 20, and 3 for GTEA, 50Salads, and Breakfast, respectively.The settings reflect different video lengths and produce similar bell-curve shapes across datasets.Source: §8 · p. 12
  • With the re-implemented ASFormer decoder, concatenated conditioning and noisy-sequence features serve as cross-attention queries and keys, while the noisy sequence supplies values.The 512-dimensional step embedding is added to the values.Source: §8 · p. 12
  • With the MS-TCN single-stage decoder, concatenated conditioning and noisy-sequence features form the input, with the step embedding added to it.Source: §8 · p. 12
  • The model omits positional encoding because it was found harmful in the original ASFormer paper, and training uses a single NVIDIA RTX 2080 GPU.Source: §8 · p. 12

9. Other Early Attempts

Early experiments tested alternative conditioning masks and combined conditional and unconditional outputs at inference. The tested mask variants were not evidently helpful, while output aggregation exposed a trade-off and difficulty detecting boundaries reliably.

9. Other Early Attempts

  • Past-feature, future-feature, and random short-clip masks were not evidently helpful in preliminary experiments.The authors describe further mask forms as a promising direction to explore.Source: §9 · p. 12
  • Combining no-mask and all-mask outputs traded edit score against accuracy, without improving both at once.An adaptive boundary-aware weighting approach was difficult because boundaries were non-trivial to detect reliably at inference.Source: §9 · p. 12
  • The authors identify further exploration as possible within their extendable framework.Source: §9 · p. 12

10. Comparison with Methods on arXiv

The supplementary comparison with recent arXiv methods does not change the paper’s main conclusion. Additional 50Salads analyses report inference and training stability and performance across video-length and action-number splits.

10. Comparison with Methods on arXiv

  • The comparison with recent arXiv methods does not change the conclusion in the main paper.The listed method using extra multi-modal features is marked as unsuitable for direct comparison.Source: §10 · p. 12, Table 9 · p. 13

11. Stability

  • Inference results on 50Salads are described as highly stable across repeated-seed evaluations.The authors distinguish macro evaluation across whole experiments from micro evaluation per video, and identify macro results as the appropriate comparison to state of the art.Source: §11 · p. 12, Table 10 · p. 13
  • Training results on 50Salads are described as stable, with narrow deviations across ten training seeds.Source: §11 · p. 12, Table 11 · p. 13

12. Effects of Video Length and Action Number

  • The method performs well across both top and bottom test-video halves divided by video length and action number.Source: §12 · p. 12, Table 12 · p. 13

13. Discussion

The discussion distinguishes DiffAct’s video-action diffusion design from diffusion image segmentation and reports that a naive modality change performs much worse. Additional visualizations show accurate, temporally coherent predictions on randomly selected videos from three datasets.

13. Discussion

  • A naive application of diffusion image segmentation to video performs much worse than the proposed task-specific design.DiffAct adapts ASFormer, uses x0 prediction, and investigates cross-entropy, smoothness, and boundary losses together.Source: §13 · p. 13

14. More Qualitative Results

  • The qualitative results are described as accurate and temporally coherent across GTEA, 50Salads, and Breakfast.Source: §14 · p. 13
  • The visualized predictions and ground truths come from randomly selected videos, with different color codes used for different datasets.Source: §14 · p. 13

15. More Results using M P at Inference

With all conditions masked at inference, DiffAct generates broadly plausible action sequences for GTEA, 50Salads, and Breakfast. The outputs exhibit dataset-specific characteristics, which the authors take as evidence that generative learning captures action-sequence priors.

15. More Results using M P at Inference

  • With all conditions masked at inference, generated action sequences are broadly plausible across the three datasets.The figures compare ground-truth sequences at the top with model results below.Source: §15 · p. 13, Figure 6 · p. 14, Figure 7 · p. 14
  • The generated sequences exhibit characteristics of each dataset, including the distinct action-class sets noted for Breakfast.Source: §15 · p. 13, Figure 8 · p. 14
  • The authors interpret these unconditional-generation results as validating the model’s ability to capture action-sequence prior distributions through generative learning.Source: §15 · p. 13
Loading 2303.17959v2…