Source-linked AI summary
Rethinking Weakly-supervised Video Temporal Grounding From a Game Perspective
Xiang Fang, Zeyu Xiong, Wanlong Fang, Xiaoye Qu, Chen Chen, Jianfeng Dong, Keke Tang, Pan Zhou, Yu Cheng, Daizong Liu
TL;DR
Weakly-supervised video temporal grounding is limited by coarse frame-query alignment and reliance on complex moment proposals. The paper uses cooperative game-theoretic interactions among frames and query words to derive query-guided frame-wise scores, achieving superior performance on Charades-STA and ActivityNet Caption, including a 4.53% gain over CPL on Charades-STA at R@5, IoU=0.5.
Problem
Existing weakly-supervised methods capture proposal- or video-level alignment without detailed frame-word consistency and rely heavily on complex moment proposals.
Method
The framework models video frames and query words as cooperative game players, learning multi-level cross-modal interactions and query-guided frame-wise relevance scores instead of using moment proposals.
Results
4.53%: Ours(Banzhaf) outperforms CPL on Charades-STA at R@5, IoU=0.5, while both variants surpass existing methods on Charades-STA and ActivityNet Captions.
Takeaways & Limitations
Game-theoretic multi-level frame-word alignment supports flexible moment localization and more accurate boundary prediction without complex moment proposals.
Takeaways & Limitations
Training requires enumerating coalition samples, increasing training time and memory; the model uses 5500 samples, after which additional samples do not affect the result.
Abstract
from arXiv · showhide
This paper addresses the challenging task of weakly-supervised video temporal grounding. Existing approaches are generally based on the moment proposal selection framework that utilizes contrastive learning and reconstruction paradigm for scoring the pre-defined moment proposals. Although they have achieved significant progress, we argue that their current frameworks have overlooked two indispensable issues: 1) Coarse-grained cross-modal learning: previous methods solely capture the global video-level alignment with the query, failing to model the detailed consistency between video frames and query words for accurately grounding the moment boundaries. 2) Complex moment proposals: their performance severely relies on the quality of proposals, which are also time-consuming and complicated for selection. To this end, in this paper, we make the first attempt to tackle this task from a novel game perspective, which effectively learns the uncertain relationship between each vision-language pair with diverse granularity and flexible combination for multi-level cross-modal interaction.Specifically, we creatively model each video frame and query word as game players with multivariate cooperative game theory to learn their contribution to the cross-modal similarity score. By quantifying the trend of frame-word cooperation within a coalition via the game-theoretic interaction, we are able to value all uncertain but possible correspondence between frames and words. Finally, instead of using moment proposals, we utilize the learned query-guided frame-wise scores for better moment localization.Experiments show that our method achieves superior performance on both Charades-STA and ActivityNet Caption datasets.
1 Introduction
Weakly-supervised video temporal grounding struggles with coarse proposal-level alignment and dependence on complex moment proposals. The paper addresses both issues with game-theoretic frame-word interaction learning and query-guided frame-wise localization.
- Weakly-supervised video temporal grounding lacks frame-wise annotations, making detailed frame-query alignment difficult.
- Existing methods generate multiple moment proposals and score their alignment with the language query, sometimes using reconstruction losses.
- Proposal-level cross-modal learning misses fine-grained frame-word alignment needed for accurate boundary grounding.
- Proposal-based performance depends heavily on proposal quality because proposals ignore each video's specific content and difficulty.
- The proposed game-theoretic framework models video frames and query words as players to learn uncertain cross-modal interactions with diverse granularity and flexible combinations.
- Experiments on Charades-STA and ActivityNet Caption show superior performance, including a 4.53% improvement over all compared methods on Charades-STA in a representative case.
2 Related Works
Fully supervised video temporal grounding typically uses annotated moment boundaries and proposal-based matching, while recent proposal-free methods directly regress temporal locations.
- Fully supervised methods receive video-sentence annotations together with corresponding moment boundaries.
- Most fully supervised approaches integrate sentence representations with predefined moment proposals and select the proposal with the highest matching score.
- Recent proposal-free methods avoid moment proposals by directly regressing the target moment's temporal locations.
3 Methodology
The methodology models video frames and query words as cooperative-game players to learn self-modal semantics and fine-grained cross-modal alignment. It then uses query-guided frame-wise scores, rather than complex moment proposals, to localize moments.
- Framework overview: The framework applies self-modal games to enhance video-frame and query-word representations before cross-modal alignment.Frame and word players contribute to modality-level semantics through game-theoretic values.
- Problem formulation: Video temporal grounding is formulated as predicting the start and end timestamps of a query-related moment from an untrimmed video.The video contains T frames, the query contains M words, and only video-level annotation is assumed.
- Self-modal game: Banzhaf values measure each frame’s weighted marginal contribution across possible coalitions of other frames.Coalitions represent consecutive or semantically similar frames, enabling contextual visual feature learning.
- Self-modal game: Learnable relevance headers provide soft supervision for game-theoretic values while reducing the complexity of direct coalition enumeration.The video header combines one-dimensional convolutions, self-attention, and decoding, while KLD aligns predicted relevance with Banzhaf values.
- Cross-modal game: Cross-modal games model cooperation between enhanced video frames and query words to learn uncertain, fine-grained alignment at multiple semantic levels.The method includes word-level interactions and extends alignment toward phrase- and sentence-level semantics.
- Training and inference: Query reconstruction conditions masked-word prediction on highlighted video frames to further enhance frame–sentence semantic relevance.The method randomly replaces one-third of query words and predicts the next word using the query prefix and highlighted-frame features.
4 Experiments
Experiments evaluate the game-based framework on Charades-STA and ActivityNet Caption, including comparisons, cross-dataset generalization, complexity, and ablations. The method achieves strong grounding performance while trading higher training cost for faster, lower-memory inference.
- Datasets and Setup: The experiments use Charades-STA and ActivityNet Caption, following prior work for video encoding and fixed feature-sequence lengths.C3D encodes ActivityNet Caption videos and I3D encodes Charades-STA videos; sequence lengths are T = 200 and T = 64, respectively.
- Quantitative Comparison: Both Banzhaf and Shapley variants surpass existing methods on Charades-STA and ActivityNet Caption.The variants use different normalization while sharing the same game interaction formulation.
- Quantitative Comparison: 4.53%: Ours(Banzhaf) outperforms CPL on Charades-STA for R@5, IoU=0.5.The paper attributes this result to learning local word-frame alignment for more accurate moment-boundary prediction.
- Generalization: Cross-dataset evaluation shows A →C performs better than C →A, while both variants outperform state-of-the-art methods such as LCNet.The paper relates this to ActivityNet Caption being larger and more complex, and to the flexibility of frame-wise alignment compared with dataset-specific proposals.
- Complexity Comparison: The method requires more training time and memory because it enumerates coalition samples, but inference is faster and uses less memory.With 4,500–5,500 samples, Banzhaf estimation error decreases from 6.9% to 1.1%; beyond 5,500 samples, additional samples do not affect the result.
- Ablation Study: Ablations show that multi-level self- and cross-modal game components, soft supervision with KLD, and reconstruction loss contribute to performance.Convolution plus self-attention captures local and global interaction; KLD outperforms MSE and CE, while removing soft supervision causes a substantial drop.
5 Conclusion
The paper concludes that game-theoretic interaction can learn uncertain video-query relationships at diverse granularities for weakly-supervised temporal grounding. Self-modal games enhance contextual semantics, while multi-level cross-modal games help determine query-related moment boundaries.
- Conclusion: The framework uses game-theoretic interaction to learn uncertain video-query relationships with diverse granularity.It introduces self-modal games for contextual enhancement and cross-modal games for fine-grained frame-word correspondence.
- Conclusion: Self-modal games correlate frames and words to enhance contextual semantics before cross-modal grounding.
- Conclusion: Multi-level cross-modal games value frame-word correspondence for more accurate query-related moment boundaries.
- Conclusion: Extensive experiments demonstrate the effectiveness of the game-based framework on two challenging datasets.