PaperlayerView exact revision on arXiv

Source-linked AI summary

Skeleton-based Zero-Shot Spatio-Temporal Action Localization via Weakly-Supervised Pretraining

Koshiro Nagano, Fumiaki Sato, Ryo Hachiuma, Kazuki Tsutsukawa, Taiki Sekii

arXiv:2608.25701v1cs.CV

TL;DR

The paper addresses the need to localize unseen actions for person instances without dense instance-level annotations or target-action training. It introduces weakly supervised vision-language pretraining with SLPS and SM-DCL, and experiments on four datasets demonstrate that the approach addresses annotation limitations while achieving competitive or superior recognition performance with lower computational and pretraining requirements.

  • Problem

    Existing spatio-temporal localization methods either depend on precise multi-person tracking or require dense frame-level, instance-level action annotations.

  • Method

    The method combines SLPS, which switches from video-level skeleton pooling during pretraining to instance-level features during inference, with SM-DCL for instance-level action discrimination in an MIL framework.

  • Results

    Experiments on four public spatio-temporal action localization and classification datasets demonstrate effective annotation-limited learning, while recognition is competitive or superior with substantially lower computational and pretraining requirements.

  • Takeaways & Limitations

    The approach identifies unseen target actions for individual instances without instance-level annotations during pretraining or multi-person tracking methods.

  • Takeaways & Limitations

    Pretraining assumes each video contains only one action, although approximately 20% of Kinetics-400 videos contain scenes with multiple distinct actions.

Abstract

from arXiv · show

We propose a novel pretraining strategy for skeleton-based zero-shot spatio-temporal action localization to estimate unseen actions for person instances while overcoming high annotation costs for training via new target actions and pretraining using large-scale action scenery datasets. Specifically, our approach, termed Skeleton-Language feature Pooling Switching, introduces a weakly-supervised vision-language pretraining mechanism. This mechanism transitions pooling kernels from pretraining, which aggregates skeleton features at the video level and aligns them with each video's known action text embeddings, to the inference phase that computes instance-level features without training via target actions. Furthermore, we propose Scene-Mixed Discriminative Contrastive Learning to distinguish actions at the instance level within the combined scene through the MIL framework. Our experiments on four public spatio-temporal action localization and classification datasets demonstrate that the proposed method effectively addresses annotation limitations.

1. INTRODUCTION

The paper targets zero-shot spatio-temporal action localization with skeletons, using weakly supervised video-level pretraining to identify unseen actions for individual instances without dense annotations or tracking.

  • Skeleton-based approaches use pose keypoints, improving robustness to appearance changes while targeting individual actions across diverse scenes.
  • Tracking-based methods depend on precise person tracking, whereas supervised localization requires dense instance-level labels for every frame.
  • Zero-shot localization avoids training on target actions but conventional methods still require considerable human annotations.
  • SLPS switches video-level skeleton pooling during pretraining to instance-level feature extraction during inference, using video-level action labels and text embeddings.
  • SM-DCL mixes instances from different scenes to distinguish individual actions under the MIL framework, addressing its single-action-per-video assumption.
  • Experiments on four spatio-temporal action localization and classification datasets evaluate the method against annotation-cost limitations.

2. RELATED WORK

Prior work spans supervised and weakly supervised localization, tracking-based skeleton methods, vision-language zero-shot recognition, and data-mixing augmentation for robustness and instance discrimination.

  • Fully supervised localization achieves high precision but requires costly frame-wise instance annotations, while weakly supervised methods use video-level labels.
  • Tracking-based methods localize actions in individual skeleton sequences generated by multi-person tracking.
  • Vision-language methods use text prompts and contrastive learning to recognize unseen visual targets, including actions at video, frame, and instance levels.
  • Data-mixing augmentation combines multiple images to improve robustness and has been applied to weakly supervised spatio-temporal action localization.

3. PROPOSED METHOD

The proposed SLPS architecture uses shared instance feature extraction while switching from video-level pooling during weakly supervised pretraining to instance-level features during inference. Its pretraining aligns pooled skeleton features with action text embeddings, while SM-DCL improves instance-level discrimination in mixed scenes.

  • Inference Phase: At inference, each instance feature is compared with target-action text embeddings using the maximum cosine similarity.An MLP aligns the dimensions of instance and text features before similarity computation.
  • Pretraining Phase: During pretraining, GMPool aggregates instance features into a video-level representation for MIL-based weak supervision.The resulting feature represents the entire video and uses only video-level action labels.
  • Pretraining Phase: The pretraining objective combines action classification and contrastive learning between video-level skeleton features and text embeddings.The classification branch uses cross-entropy over action classes, while the contrastive branch uses a projection head and symmetric video-to-text and text-to-video losses.
  • SLPS DNN Architecture: SLPS extracts skeleton features for each instance in both phases, then changes the aggregation unit between pretraining and inference.GMPool produces video-level features during pretraining, whereas inference retains instance-level features for target-action scoring.
  • SM-DCL: SM-DCL mixes multiple scenes, restores the backbone output to the unmixed state before pooling, and enhances instance-level discriminability with minimal computational overhead.The method addresses the assumption that a video contains only one action class when multiple people interact.

4. EXPERIMENTS

Experiments evaluate the proposed method across spatio-temporal localization and violent-action classification settings, comparing it with weakly supervised, tracking-based, supervised, and appearance-based approaches. The method outperforms several conventional baselines while achieving competitive recognition with substantially lower computational and pretraining requirements.

  • Datasets and evaluation settings: The experiments compare performance on UCF101-24, FDD, RWF-2000, and MF using localization and classification benchmarks.UCF101-24 evaluates weakly supervised localization, FDD evaluates tracking-based localization, and RWF-2000 and MF evaluate violent-action classification.
  • Comparative experiments: The proposed method outperforms the conventional weakly-supervised method on UCF101-24 spatio-temporal action localization.This result holds although UCF101-24 is described as advantageous for appearance-based approaches.
  • Comparative experiments: The proposed method considerably outperforms conventional tracking-based methods on FDD spatio-temporal action localization.The comparison uses the proposed method with less accurate pose detectors than those used by the conventional methods.
  • Comparative experiments: The proposed method outperforms many supervised methods on violent-action classification and achieves accuracy comparable to SPIL and existing supervised SoTA architectures.The reported classification results use outputs from the proposed spatio-temporal localization method.
  • Summary: The method achieves zero-shot spatio-temporal action localization without target action-specific DNN training, using only video-level annotations during pretraining.The paper reports that this reduces annotation costs for pretraining and training new target actions.
  • Efficiency comparison: The model contains about 70M parameters, uses only Kinetics-400 for training, and reaches approximately 1900 FPS versus Qwen2.5-VL-7B-Instruct at approximately 8 FPS.The reported comparison corresponds to about a 240× speedup and substantially lower computational and pretraining requirements.

5. CONCLUSION

The paper introduces skeleton-based zero-shot spatio-temporal action localization to estimate unseen actions for person instances while addressing annotation limitations. Its weakly supervised pretraining and instance-level contrastive learning target localization without instance-level pretraining annotations or multi-person tracking.

  • The method estimates unseen actions for person instances detected in each video frame, addressing annotation limitations in existing localization methods.
  • Weakly supervised pretraining learns from video-level action labels without training on target actions.
  • The approach identifies actions for each instance during inference without instance-level annotations during pretraining or multi-person tracking.
  • Scene-Mixed Discriminative Contrastive Learning distinguishes actions at the instance level within the Multiple Instance Learning framework.
  • Experiments evaluate the method’s effectiveness against annotation limitations.

6. EXPERIMENTS

The experiments cover four action datasets and evaluate the proposed skeleton-based approach through dataset-specific protocols, pooling components, qualitative localization, and pretraining settings.

  • Datasets: Kinetics-400 provides 250K training and 19K validation 10-second clips across 400 action classes for pretraining.
  • Datasets and evaluation: UCF101-24 contains 3.2K videos spanning 24 action classes and is evaluated with Video AP (%) at 3D IoU=0.5.
  • Architecture: The Grouped Pool Block combines global GMPool features with local Max-Pooling features for each instance group.
  • Pooling switching: During pretraining, GMPool aggregates instance-level features into a video-level feature, whereas inference removes GMPool through pooling switching.
  • Qualitative evaluation: Qualitative results show spatio-temporal localization on UCF101-24 without instance-level training annotations or multi-person tracking.
  • Training setup: The study uses spatial and temporal joint augmentations, including coordinate transforms, temporal cropping, and random joint dropping, with 150 plus 40 pretraining epochs.

6.4. Ablation Studies

The ablation studies examine assumptions, scene mixing, text encoders and prompts, dataset scale, loss design, pooling switching, and qualitative behavior across action settings.

  • Assumptions: The single-action-per-video pretraining assumption covers approximately 80% of Kinetics-400 scenes, while added skeleton noise causes only 1.2% points accuracy degradation on UCF101-24.
  • Qualitative results: Qualitative results include violent-behavior classifications on RWF-2000 and zero-shot localization of fall behaviors on FDD.
  • Component ablations: The ablations separately evaluate action-classification and contrastive-loss mixing ratios, pooling switching, and contrastive-learning loss functions on UCF101-24.
  • Outlier actions: The paper reports average accuracy for three UCF101-24 outlier actions with the lowest similarity to pretraining actions.
  • SM-DCL: SM-DCL uses two mixed scenes because experiments over scene counts and mixing timings selected the setting with the highest accuracy.
  • Pretraining scale: UCF101-24 accuracy increases as progressively larger Kinetics-400 subsets are used for pretraining.
  • Text representations: The method remains effective with different text encoders and localizes unknown actions with some accuracy across varied text prompts on RWF-2000.
  • Comparative evaluation: The summary table compares spatio-temporal action localization approaches.

7. HUMAN POSE DETECTORS

The paper describes two human pose detectors used to obtain skeleton inputs: bottom-up PPNs and top-down HRNet.

  • PPNs: PPNs detect human skeletons bottom-up at high speed using a ResNet-101 backbone trained on MS-COCO.
  • HRNet: HRNet is a top-down human pose detector described as achieving superior accuracy, with computational cost included in the detector setup.

8. QUALITATIVE RESULTS ON THE FDD DATASET

On the FDD dataset, the proposed method qualitatively localizes falling actions for each person in every frame without training on the target action.

  • The proposed method qualitatively localizes falling actions for each person in every frame on the FDD dataset.The localization is performed in a zero-shot manner.
  • FDD qualitative results demonstrate person-level action localization across individual video frames.
  • The reported result concerns falling actions detected for each person instance rather than only the entire video.

9. RELATED WORK

Related work spans action classification, temporal localization, and spatio-temporal localization. This study focuses on skeleton-based spatio-temporal localization, recognizing actions for each detected skeleton in every frame.

  • The related-work discussion compares the proposed method's features with those of prior works in Table 15.
  • Action recognition research commonly addresses classification, temporal localization, and spatio-temporal localization.
  • The study focuses on skeleton-based spatio-temporal action localization.Actions are recognized for each skeleton detected in every frame.
Loading 2608.25701v1…