Source-linked AI summary
SF-Net: Single-Frame Supervision for Temporal Action Localization
Fan Ma, Linchao Zhu, Yi Yang, Shengxin Zha, Gourab Kundu, Matt Feiszli, Zheng Shou
TL;DR
Temporal action localization needs temporal information, but full boundary annotation is costly and weak video-level supervision provides limited temporal cues. SF-Net uses single-frame annotations with actionness prediction and pseudo-frame mining, significantly outperforming weakly supervised methods across segment and single-frame localization on three benchmarks. The results show that this low-cost supervision can support TAL performance approaching fully supervised methods.
Problem
Full temporal-boundary annotation is resource intensive, while weak video-level supervision provides less temporal information for localizing actions.
Method
SF-Net predicts per-frame actionness and mines pseudo action and background frames from single-frame annotations and unlabeled frames for joint classifier training.
Results
SF-Net significantly outperforms weakly supervised methods in both segment localization and single-frame localization on three standard benchmarks.
Takeaways & Limitations
Single-frame supervision can support temporal action localization while requiring substantially fewer annotation resources than full supervision.
Abstract
from arXiv · showhide
In this paper, we study an intermediate form of supervision, i.e., single-frame supervision, for temporal action localization (TAL). To obtain the single-frame supervision, the annotators are asked to identify only a single frame within the temporal window of an action. This can significantly reduce the labor cost of obtaining full supervision which requires annotating the action boundary. Compared to the weak supervision that only annotates the video-level label, the single-frame supervision introduces extra temporal action signals while maintaining low annotation overhead. To make full use of such single-frame supervision, we propose a unified system called SF-Net. First, we propose to predict an actionness score for each video frame. Along with a typical category score, the actionness score can provide comprehensive information about the occurrence of a potential action and aid the temporal boundary refinement during inference. Second, we mine pseudo action and background frames based on the single-frame annotations. We identify pseudo action frames by adaptively expanding each annotated single frame to its nearby, contextual frames and we mine pseudo background frames from all the unannotated frames across multiple videos. Together with the ground-truth labeled frames, these pseudo-labeled frames are further used for training the classifier. In extensive experiments on THUMOS14, GTEA, and BEOID, SF-Net significantly improves upon state-of-the-art weakly-supervised methods in terms of both segment localization and single-frame localization. Notably, SF-Net achieves comparable results to its fully-supervised counterpart which requires much more resource intensive annotations. The code is available at https://github.com/Flowerfan/SF-Net.
1 Introduction
Single-frame supervision provides more temporal information than video-level labels while requiring far less annotation effort than full temporal boundaries. SF-Net exploits these annotations to mine pseudo action and background frames, predict actionness, and improve temporal action localization.
- Motivation: Full supervision requires precise action boundaries, whereas weak supervision records only video-level action classes.Weak annotation avoids repeated viewing and substantially reduces annotation resources.
- Motivation: Single-frame supervision records one timestamp for each action instance, preserving temporal cues with low annotation overhead.The annotator records the action class and timestamp when noticing each action.
- SF-Net: SF-Net addresses unresolved temporal-boundary localization by predicting per-frame actionness and mining pseudo background and action frames.These innovations improve discrimination between action and background frames for TAL.
- Evaluation: Single-frame localization evaluates whether one frame per action instance can be localized when precise start and end times are unnecessary.The paper motivates this setting with applications such as retrieving accident frames from street-camera archives.
- Contributions: The paper presents single-frame supervision as the first use of this annotation type for temporal action-boundary localization and reports reduced annotation time versus full supervision.The contribution explicitly combines lower annotation cost with the challenging boundary-localization problem.
- Results: SF-Net significantly improves both segment localization and single-frame localization across three benchmarks.The experiments report gains on both tasks, though the supplied passage does not provide numerical values.
2 Related Work
Prior work spans action recognition, fully supervised TAL, weakly supervised TAL, and point supervision. SF-Net is framed as applying single-frame supervision specifically to temporal action localization.
- Action recognition: Action-recognition research uses temporal and spatial information, including handcrafted features and deep networks such as two-stream models.The supplied passage positions this literature as broader action-recognition work rather than the paper’s central supervision setting.
- Action recognition: Single-frame supervision is distinguished from self-supervised and few-shot feature learning because this paper targets temporal action localization.The related-work passage explicitly narrows the paper’s focus to single-frame supervision for TAL.
- Point supervision: Point supervision has been extended from image segmentation to video spatio-temporal localization, while single-frame supervision uses only one action frame.The paper characterizes per-frame spatial-point annotation as excessive for temporal localization.
- Fully supervised TAL: Fully supervised TAL commonly follows proposal generation followed by classification using temporal boundary annotations.Other studied categories include sequential decision-making and single-shot detectors.
- Weakly supervised TAL: Weakly supervised TAL often uses multiple-instance learning and temporal action-score sequences to generate proposals without boundary annotations.The supplied passage identifies MIL as a widely used approach in this setting.
- SF-Net: SF-Net combines frame mining with classification and actionness modules to use single-frame supervision for TAL.The framework mines pseudo action and background frames and jointly trains both modules.
3 Method
SF-Net trains temporal action localization from single-frame supervision by combining frame/video classification, actionness prediction, and pseudo-label mining. At inference, classification and actionness scores are thresholded to produce action segments.
- Problem Definition: SF-Net defines segment localization and single-frame localization as its two temporal localization tasks.Segment localization predicts action boundaries and classes; single-frame localization predicts one timestamp and class per detected instance.
- Framework: The framework jointly trains classification and actionness modules using labeled frames, video-level labels, and mined pseudo frames.The classification module produces per-frame and video-level scores, while the actionness module predicts each frame’s probability of containing an action.
- Pseudo Label Mining: Action-frame mining expands each annotated frame to nearby frames when predicted labels and class-score criteria remain consistent.Expansion proceeds separately into past and future frames and is limited by a predefined radius.
- Pseudo Label Mining: Background-frame mining selects likely background frames from unlabeled frames across videos and uses them to train an explicit background class.The method ranks unlabeled frames by their background-class scores and selects the top ηK frames.
- Training Objectives: The training objective combines frame classification, video classification, and actionness losses with weighting hyperparameters α and β.The frame classification loss includes annotated action frames and mined background frames.
- Inference: During inference, consecutive frames retained after thresholding the classification and actionness scores form detected action segments.The classification module identifies possible action categories, while actionness estimates whether frames contain target actions.
4 Experiment
Experiments evaluate SF-Net on THUMOS14, GTEA, and BEOID using segment and single-frame localization metrics, annotation studies, ablations, and comparisons with prior methods. Results show that single-frame annotations substantially reduce annotation time while improving localization over weak supervision, with dataset-dependent effects from background and action-frame mining.
- Datasets and metrics: Experiments use THUMOS14, GTEA, and BEOID, evaluating segment localization with mAP across IoU thresholds and single-frame localization with mAP@hit.For single-frame localization, a prediction is correct when it falls within the ground-truth action segment and has the correct class label.
- Annotation analysis: Human annotators show low variance in the number of labeled action instances, with annotated single frames nearly matching the ground-truth segment count.The annotation analysis reports that annotators rarely miss action instances despite pausing only once per action.
- Annotation analysis: Annotators tend to select frames near the middle of action segments, while rare annotations fall outside the corresponding ground-truth temporal range.The distributions are shown across datasets using relative action position on the x-axis and annotated-frame percentage on the y-axis.
- Annotation analysis: 45s, 50s, and 300s are the average times per video minute for video-level, single-frame, and segment labels, respectively.Single-frame annotation therefore takes nearly the same time as video-level labeling and much less time than fully supervised temporal annotation.
- Ablation analysis: Single-frame supervision outperforms weak supervision, while background and action-frame mining improve performance on BEOID and THUMOS14.On GTEA, performance decreases with background mining because the dataset contains almost no background frames; the actionness module is reported to work well on BEOID and THUMOS14.
- Comparison with state of the art: SF-Net outperforms previous weakly supervised methods at all evaluated IoU thresholds on THUMOS14, despite using only one annotated frame per action instance.The comparison includes weakly supervised and fully supervised methods, and also examines human-annotated versus sampled single frames.
5 Conclusions
SF-Net leverages single-frame supervision for temporal action localization while supporting both segment and single-frame localization. The study also uses controlled annotation procedures to improve label consistency and quality.
- Contributions: SF-Net predicts actionness and mines pseudo background and action frames to exploit single-frame supervision.The system is designed for both segment localization and single-frame localization during inference.
- Results: SF-Net significantly outperforms weakly-supervised methods on segment and single-frame localization across three standard benchmarks.
- Annotation: Four annotators label one frame for each target action instance after training on action classes and examples.The annotation protocol includes textual definitions, positive single-frame examples, and segmented action instances.
A.2 Annotation tool
The annotation tool streamlines single-frame labeling by recording timestamps automatically and saving annotations after each video. Multiple annotators and training procedures are used to reduce ambiguity and annotation bias.
- Annotation workflow: The tool automatically records timestamps, supports deleting incorrect annotations, and saves completed annotations to a CSV file.Annotators can generate the saved records after watching the whole video.
- Annotation workflow: Annotators pause the video, select the target action class, and resume playback to label subsequent action instances.The interface records the paused frame and its selected class as part of the annotation workflow.
- Quality control: Each video is labeled by four annotators, with randomly selected annotations used during experiments to reduce annotation bias.Annotators are trained to recognize target actions while watching the video.
B Action Frame Mining
Action frame mining expands each labeled action frame into nearby pseudo-labeled frames using classification scores and neighboring-label consistency. The evaluation compares classification accuracy with class-agnostic localization performance.
- Action Frame Mining: Action frame mining treats each labeled action frame as an anchor and expands it to nearby frames when score and neighbor-consistency conditions hold.Expanded frames receive the anchor frame’s action label.
- Action Frame Mining: Algorithm 1 uses a video classification activation, labeled frame, action class, expansion radius r = 5, and threshold ξ = 0.9 to produce an expanded frame set.Frames are added when neighboring predictions agree and the current action score reaches the threshold relative to the anchor.
- Evaluation: The evaluation reports video-level classification mAP and class-agnostic localization AP at different IoU thresholds.Classification and localization are evaluated independently.
D.1 Background Ratio
The background-ratio experiments evaluate segment and frame metrics across repeated runs on THUMOS14. Background mining improves both metric types, with stability across η values from 3 to 9, while loss-coefficient results indicate limited hyper-parameter sensitivity.
- Background Ratio: Background mining boosts segment and frame evaluation metrics on THUMOS14.The experiments report means and standard deviations over three runs with randomly sampled annotations.
- Background Ratio: The model becomes stable when the background ratio η ranges from 3 to 9.
- Loss coefficients: The model is not sensitive to the evaluated loss hyper-parameters α and β.The default values of α and β are 1, and each coefficient is varied while the other is fixed.
E ActivityNet Simulation Experiment
On ActivityNet1.2, SF-Net is evaluated with randomly sampled single-frame annotations and reports segment localization using mAP across IoU thresholds. The method retains a performance gain under this simulated supervision setting.
- AVG reports the average mAP from IoU 0.5 to 0.95.
- SF-Net is evaluated on the ActivityNet1.2 validation set using single frames randomly sampled from ground-truth segments.The experiment follows the standard evaluation protocol.
- SF-Net obtains a performance gain with single-frame supervision on this large-scale dataset.
F Qualitative Results
On BEOID, SF-Net localizes and classifies action instances correctly, with temporal boundaries close to ground truth despite lacking boundary annotations during training. However, it can produce more detected segments per class than the ground truth.
- SF-Net localizes every action instance and assigns each instance the correct action category.The first example contains scan card and open door actions.
- The predicted temporal boundaries are close to ground truth despite no temporal boundary information during training.
- SF-Net detects all positive instances in the second example but produces more segments per action class than the ground truth.The authors identify fine-grained action encoding as future work for distinguishing action classes.