Source-linked AI summary
Weakly-Supervised Action Localization with Expectation-Maximization Multi-Instance Learning
Zhekun Luo, Devin Guillory, Baifeng Shi, Wei Ke, Fang Wan, Trevor Darrell, Huijuan Xu
TL;DR
Weakly-supervised action localization must identify action clips from video-level labels, but attention-MIL learns instance weights as a by-product of bag classification. This paper treats key-instance assignment as latent and alternates EM pseudo-labeling updates, reporting state-of-the-art performance.
Problem
With only video-level labels, action localization requires assigning which clips trigger each bag label, while attention-MIL learns instance weights indirectly through bag classification.
Source: §1 · pp. 1–2Method
The method models key-instance assignment as a hidden variable and uses two pseudo-label schemes to alternate the EM assignment and classification steps.
Source: §1 · p. 3Results
The EM-MIL framework achieves state-of-the-art performance.
Source: §1 · p. 3, §5 · p. 14Takeaways & Limitations
Hard pseudo-labels maintain high recall across EM iterations and increase proposal completeness by delaying focus on discriminative action parts.
Source: §3.4 · p. 9The method is framed under binary MIL, where a negative bag requires every instance to be negative.
Source: §3 · p. 5
Abstract
from arXiv · showhide
Weakly-supervised action localization requires training a model to localize the action segments in the video given only video level action label. It can be solved under the Multiple Instance Learning (MIL) framework, where a bag (video) contains multiple instances (action segments). Since only the bag's label is known, the main challenge is assigning which key instances within the bag to trigger the bag's label. Most previous models use attention-based approaches applying attentions to generate the bag's representation from instances, and then train it via the bag's classification. These models, however, implicitly violate the MIL assumption that instances in negative bags should be uniformly negative. In this work, we explicitly model the key instances assignment as a hidden variable and adopt an Expectation-Maximization (EM) framework. We derive two pseudo-label generation schemes to model the E and M process and iteratively optimize the likelihood lower bound. We show that our EM-MIL approach more accurately models both the learning objective and the MIL assumptions. It achieves state-of-the-art performance on two standard benchmarks, THUMOS14 and ActivityNet1.2.
1 Introduction
Weakly-supervised action localization seeks action clips from video-level labels, but requires assigning which clips trigger each bag’s label. The paper uses EM to model that assignment explicitly and reports state-of-the-art performance.
1 Introduction
- Video-level labels avoid costly frame-level start and end annotations, but leave the key-instance assignment within each video unresolved.Source: §1 · pp. 1–2
- Attention-MIL learns instance weights as a by-product of bag classification, and the authors link this objective mismatch to incomplete action proposals.The attention module may focus only on discriminative action parts to classify a bag.Source: §1 · p. 2
- The authors report state-of-the-art performance, including on ActivityNet1.2.Source: §1 · p. 3
- EM treats key-instance assignment as a hidden variable and uses two pseudo-label schemes to alternate assignment and classification updates.The alternating updates optimize a lower bound of the MIL objective and address the treatment of negative bags.Source: §1 · p. 3
2 Related Work
Prior approaches include attention-based action localization and iterative refinement, while traditional MIL offers a precedent for modeling key instances as hidden variables. The paper distinguishes its EM approach through alternating updates within one model.
2 Related Work
- Attention-based action methods use clip contributions or features to form video-level predictions, while erasing, subsampling, and multi-branch approaches address incomplete localization.Source: §2 · p. 3
- The architecture uses fixed two-stream I3D features and alternates updates to the key-instance assignment and classification branches, using each branch to generate pseudo-labels for the other.Source: Figure 2 · p. 4
- Unlike pass-to-pass refinement methods, EM alternates pseudo-labeling and training between modules of the same model, requiring one pass.Source: §2 · p. 4
- Traditional MIL methods modeled key-instance assignment iteratively, but preceded neural networks and were not applied to high-dimensional action localization.Source: §2 · pp. 4–5
3 Method
EM-MIL treats key-instance assignment as latent and alternates pseudo-label updates for instance assignment and classification. Its hard, foreground-focused labeling is designed around instance localization and the MIL assumption that negative bags contain only negative instances.
3 Method
- Videos are MIL bags of clips, with video-level labels observed and latent binary assignments marking which clips generate each class label.The video label is generated from assigned clips’ class probabilities through a permutation-invariant operator.Source: §3 · p. 5
3.1 E Step
- E-step pseudo-labels train qφ from clip-classification scores, marking a clip positive when its score for any present class exceeds the video’s temporal mean.The resulting labels update qφ with binary cross-entropy.Source: §3.1 · p. 6
3.2 M Step
- M-step pseudo-labels train pθ from qφ scores, marking clips positive above a video-relative threshold and negative otherwise, including clips in negative bags.The threshold includes γ times the range of assignment scores; γ encodes a prior on how similarly an action appears across videos.Source: §3.2 · p. 7
3.3 Overall Algorithm
- The overall algorithm alternates: E-step updates qφ from pθ-derived pseudo-labels, then M-step updates classification from qφ, optimizing video-label likelihood.The two steps proceed until convergence.Source: §3.3 · p. 7
3.4 Comparison with Previous Methods
- Unlike prior attention-MIL, EM-MIL applies its threshold-based attention only in positive bags, aligning with the assumption that all instances in negative bags are negative.The figures contrast foreground-only pseudo-labeling and key-instance assignment with prior methods’ all-class attention.Source: §3.4 · p. 8, Figure 3 · p. 8, Figure 4 · p. 8
- Hard pseudo-labels initially mark every clip in positive bags positive, then narrow the search, maintaining high action-clip recall across EM iterations and increasing proposal completeness.The paper also frames its instance-level localization objective as more aligned with the task than prior bag-level attention-MIL objectives.Source: §3.4 · p. 9
3.5 Inference
- At inference, the localization score fuses assignment and classification scores; thresholded clips are grouped into temporal action proposals.The fusion weight is set by dataset, and proposals use the thresholding scheme described for the M-step.Source: §3.5 · p. 9
4 Experiments
Experiments evaluate EM-MIL on THUMOS14 and ActivityNet1.2, reporting state-of-the-art mAP@0.5 on both; ablations also compare the proposed pseudo-labeling and alternating EM training.
4.1 Experimental Setup
- THUMOS14 validation videos are used for training and test-set evaluation, while ActivityNet1.2 results use validation videos; localization is measured by mAP@α at different tIoU thresholds.Average mAP over ten evenly distributed thresholds from 0.5 to 0.95 is also commonly reported.Source: §4.1 · p. 10
4.2 Comparison with State-of-the-art Approaches
- 30.5% mAP@0.5 on THUMOS14 yields a 3% improvement over the best of six recent same-backbone models, while 37.4% mAP@0.5 on ActivityNet1.2 surpasses published models.The THUMOS14 result is reported without auxiliary regularization losses.Source: Table 1 · p. 11, §4.2 · pp. 11–12, Table 2 · p. 13
- THUMOS14 shows stronger high-tIoU but slightly weaker low-IoU results, whereas ActivityNet1.2 performs worse at high tIoU; the authors speculate classification is its bottleneck.They contrast THUMOS14’s multiple short action segments with ActivityNet1.2’s mostly single-class videos with few long segments, and use different localization-score weights for the two datasets.Source: §4.2 · pp. 11–13
4.3 Ablation Studies
- 30.5% versus 24.5% mAP@0.5 for Full Model and the alternating model with plain attention, respectively, supports the proposed pseudo-labeling strategy.The authors say the strategy explicitly models key-instance assignment and better aligns with the MIL assumption.Source: Table 3 · p. 14, §4.3 · p. 14
- 30.5% versus 26.8% mAP@0.5 for Full Model and the joint-optimization pseudo-labeling model, respectively, supports alternating EM training.The EM procedure iteratively estimates key-instance assignment and then maximizes video classification accuracy.Source: Table 3 · p. 14, §4.3 · p. 14
5 Conclusion
The paper proposes EM-MIL with pseudo-labeling and alternating training for weakly supervised action detection. Explicitly modeling latent variables gives greater control over the instance-level MIL objective, while the conclusion identifies more sophisticated EM methods and temporal structure as future directions.
5 Conclusion
- EM-MIL explicitly models latent variables to improve control over the instance-level MIL learning objective.The framework uses pseudo-labeling and alternating training for weakly supervised action detection.Source: §5 · p. 14
- The framework is motivated by traditional MIL literature, which the authors describe as under-explored in deep learning settings.Source: §5 · p. 14
- The authors report that EM-MIL leads to state-of-the-art performance.Source: §5 · p. 14
- Future directions include more sophisticated EM methods and incorporating video temporal structure to improve performance.The authors describe their current pseudo-labeling scheme as relatively simple.Source: §5 · p. 14