PaperlayerView exact revision on arXiv

Source-linked AI summary

CoDy: Codec Dynamics as Complementary Priors for Unsupervised Temporal Action Localization

Tianyu Yan, Qingyi Yang, Qin Yang, Xufei Guo, Yuan Tang, Rui Wang, Yixue Hao, Long Hu

arXiv:2610.19880v1cs.CV

TL;DR

UTAL must infer action categories and temporal boundaries without annotation, while existing methods rely mainly on RGB and optical flow. CoDy adds codec dynamics as complementary priors for category discovery and boundary learning. On THUMOS14 and ActivityNet v1.3, it achieves state-of-the-art performance.

  • Problem

    UTAL must jointly discover latent action categories and localize temporal extents without annotations, while visual-only evidence can yield noisy categories and inaccurate boundaries.

    Source: §1 · pp. 1–2

  • Method

    CoDy combines a codec-enhanced view with RGB and flow for confidence-aware consensus clustering, and uses codec dynamics as class-agnostic priors for visual boundary learning and proposal refinement.

    Source: §1 · p. 2

  • Results

    CoDy achieves state-of-the-art performance on THUMOS14 and ActivityNet v1.3, with pronounced gains at high IoU thresholds.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Codec dynamics provide complementary annotation-free priors for jointly learning action categories and temporal boundaries.

    Source: §1 · p. 2, §6 · p. 8

Abstract

from arXiv · show

Unsupervised temporal action localization (UTAL) aims to identify action categories and localize their temporal boundaries without annotation supervision. Existing methods typically derive supervision for both tasks solely from RGB and optical-flow representations, often resulting in noisy category assignments and inaccurate temporal boundaries. In this work, we reveal an overlooked property of video codecs: group-of-pictures quantization patterns preserve category-discriminative dynamics, while motion-vector and residual variations exhibit pronounced responses around action transitions. Based on this insight, we propose CoDy, a codec dynamics-guided framework that exploits the codec as complementary information to the visual view for UTAL. CoDy integrates a codec-enhanced view with RGB and flow through confidence-aware consensus clustering, producing reliable soft pseudo-labels. It further converts codec dynamics into a class-agnostic boundary prior, which supervises visual boundary learning and guides proposal initialization and iterative refinement. The codec branches are only used during the training stage and introduce no extra codec-side computation at inference. Extensive experiments on THUMOS14 and ActivityNet v1.3 demonstrate that CoDy consistently outperforms existing UTAL methods and achieves state-of-the-art localization performance. These results establish codec signals as effective annotation-free priors for jointly inferring which actions occur and when they occur.

1. Introduction

UTAL must discover latent action categories and localize their extents without labels, but visual evidence can make both objectives unreliable and mutually reinforcing. CoDy uses codec dynamics as complementary priors for category discovery and boundary localization, achieving state-of-the-art results on two benchmarks.

1. Introduction

  • UTAL jointly discovers latent action categories and temporal extents without annotations, with errors in discovery and localization able to amplify one another.The task assumes only that the number of latent action categories is known.Source: §1 · p. 1
  • RGB and flow can produce correlated errors: scene context and camera motion weaken category structure, while representations favoring action interiors can yield incomplete or over-extended boundaries.Iteration may reinforce rather than correct these errors, motivating complementary evidence.Source: §1 · p. 2
  • Codec dynamics offer complementary annotation-free cues: GOP quantization patterns preserve category-discriminative dynamics, while motion-vector and residual variations respond around action transitions.These cues do not directly specify action categories or exact boundaries.Source: §1 · p. 2
  • CoDy combines a GOP-level codec view with RGB and flow through confidence-aware consensus clustering, then uses codec variations as class-agnostic boundary priors for temporal refinement.Codec-derived proposals also feed back to refine category assignments; codec processing is used only during training.Source: §1 · p. 2
  • CoDy establishes state-of-the-art performance on THUMOS14 and ActivityNet v1.3, with particularly pronounced gains at high IoU thresholds.The reported gains support codec priors for both pseudo-label generation and boundary-prior learning.Source: §1 · p. 2

2. Related Work

UTAL methods iteratively infer latent categories and use pseudo-labels to train localization, while codec research has mainly treated compressed signals as alternatives or complements for video understanding. These lines of work frame category reliability and boundary localization as challenges for UTAL, and leave codec predictive structure as a potential prior.

2.1. Unsupervised Temporal Action Localization

  • UTAL methods typically cluster representations into latent categories, use pseudo-labels for localization learning, and update representations for subsequent clustering; noisy assignments can accumulate across iterations.Background content, intra-class variation, and similar actions can make category assignments noisy and weaken localization supervision.Source: §2.1 · pp. 2–3

2.2. Codec Representations for Video Understanding

  • Compressed-domain studies use codec signals as efficient alternatives or complements to decoded visual features, including for action recognition and video-language modeling.Examples include CoViAR [21], MM-ViT [3], and CoPE-VideoLM [4].Source: §2.2 · p. 3

3. Empirical Analysis of Codec-Prior Dynamics

CoDy evaluates codec dynamics as complementary cues for category discovery and boundary localization, using GOP observations at 0.5-second resolution.

3. Empirical Analysis of Codec-Prior Dynamics

  • Videos were re-encoded into fixed-length 0.5-second GOPs, each yielding a temporally aligned codec observation with spatial structure and temporal evolution.The observation includes an I-frame and the motion-vector and residual fields of its P-frames.Source: §3 · p. 3

3.1. Observation 1: Codec Dynamics Complement Category Discrimination

  • Codec augmentation raised pseudo-label accuracy from 60.0% to 63.5% and macro-F1 from 52.87% to 56.16% in controlled clustering comparisons.The codec view also reduced confusion between visually similar JavelinThrow and ThrowDiscus actions.Source: §3.1 · pp. 3–4

3.2. Observation 2: Codec Dynamics Exhibit Boundary Sensitivity

  • Codec responses showed sharper, more consistent elevations around action boundaries than RGB and flow, providing a more sensitive class-agnostic transition cue.Boundary annotations were used only for evaluation, not to construct the responses.Source: §3.2 · p. 4

4. Method

CoDy adds codec-derived signals to a proposal-based UTAL framework to support latent category discovery and temporal localization. It uses codec information to generate pseudo-labels and boundary priors, then refines proposals and category predictions.

4.1. Framework Overview

  • CoDy builds on P-MIL with codec-guided branches for pseudo-label generation and boundary-prior learning, iteratively refining category predictions and temporal boundaries.The model predicts action instances with temporal boundaries, discovered categories, and confidence scores.Source: §4.1 · pp. 4–5, §4 · p. 4

4.2. Codec-Enhanced Pseudo-Label Generation

  • The codec view combines I-frame appearance tokens with P-frame motion-vector and residual tokens, then aggregates GOP sequences into a video-level representation for clustering.The GOP representation contains 8 + 8n_t codec tokens for n_t P-frames.Source: §4.2 · p. 5
  • Confidence-aware clustering fuses normalized visual and codec views, weighting each view by its distance to a cluster center to produce soft pseudo-labels.The pseudo-label distribution is periodically updated as visual and codec representations evolve during training.Source: §4.2 · p. 5

4.3. Codec-Based Boundary Prior Learning

  • Parallel multi-scale networks process snippet-level visual features and GOP-level codec features; aligned codec predictions provide a class-agnostic prior for visual boundary learning.High-response codec timestamps are paired under duration constraints to initialize proposals, reducing unreliable early candidates and supporting later refinement.Source: §4.3 · pp. 5–6

4.4. Training and Inference

  • Training uses soft pseudo-labels for classification and jointly optimizes visual and codec boundary branches, while iterative proposal ranking removes low-scoring candidates and adds boundary-based proposals.At inference, pseudo-label generation and the codec prior branch are discarded; the learned visual boundary branch refines baseline proposals for a fixed number of rounds.Source: §4.4 · p. 6

5. Experiments

CoDy is evaluated on THUMOS14 and ActivityNet v1.3, where it outperforms existing unsupervised methods at most IoU thresholds and shows especially pronounced gains at stricter thresholds. Ablations indicate that codec cues support both proposal refinement and pseudo-label quality, with motion vectors and residuals serving different roles.

5.1. Experimental Setup

  • The evaluation uses THUMOS14 and ActivityNet v1.3, reporting detection mAP across IoU thresholds for each benchmark.THUMOS14 is evaluated at thresholds from 0.1 to 0.7; ActivityNet v1.3 reports mAP at 0.5, 0.75, 0.95, and the average over 0.5–0.95.Source: §5.1 · p. 6

5.2. Main Results

  • CoDy outperforms existing unsupervised methods at most IoU thresholds on THUMOS14 and ActivityNet v1.3, with particularly pronounced gains at stricter thresholds.The reported results associate the stricter-threshold gains with improved temporal boundary precision.Source: §5.2 · p. 7, Table 1 · p. 7, Table 2 · p. 7

5.3.1. Impact of Key Components.

  • Codec-enhanced pseudo-label generation and codec-prior boundary guidance each improve performance over the codec-free baseline, and combining them yields the highest accuracy.The semantic view enhances pseudo-label quality, while the boundary prior improves proposal temporal precision.Source: §5.3.1 · p. 7, Table 3 · p. 7

5.3.2. Impact of Proposal Sources.

  • Codec proposals begin at 10.60% average mAP, versus RGB at 5.78%, flow at 5.76%, and RGB-flow fusion at 5.79%.Video-level category labels are supplied to all variants to isolate proposal quality; codec proposals maintain higher accuracy during refinement.Source: §5.3.2 · p. 8, Figure 4 · p. 8

5.3.3. Impact of Codec View for Pseudo-Label Quality.

  • Adding the codec view produces more accurate and balanced pseudo-labels before model iteration and maintains an advantage across successive refreshes.The comparison evaluates generated labels using aligned accuracy and macro-F1 after Hungarian cluster-to-category assignment.Source: §5.3.3 · p. 8, Figure 4 · p. 8

5.3.4. Impact of Motion Vector and Residual Features.

  • Motion vectors are more effective for classification, while residuals perform better for boundary modeling around action transitions.The passage identifies explicit motion cues as useful for semantic understanding and residuals as preserving local appearance variations and temporal discontinuities.Source: §5.3.4 · p. 8, Table 4 · p. 8

6. Conclusion

CoDy uses video-bitstream information as complementary supervision for unsupervised temporal action localization. Experiments on THUMOS14 and ActivityNet v1.3 demonstrate superior performance, supporting codec dynamics as a source for learning action categories and temporal boundaries.

  • CoDy combines a codec view with RGB and flow for category discovery and pseudo-label generation, while motion-vector and residual dynamics guide boundary learning and proposal refinement.The conclusion presents codec dynamics as complementary supervision for learning action categories and temporal boundaries.Source: §6 · p. 8
Loading 2610.19880v1…