Source-linked AI summary
Annotations Are Not All You Need: A Cross-modal Knowledge Transfer Network for Unsupervised Temporal Sentence Grounding
Xiang Fang, Daizong Liu, Wanlong Fang, Pan Zhou, Yu Cheng, Keke Tang, Kai Zou
TL;DR
The paper addresses unsupervised temporal sentence grounding without paired video-query or segment-boundary annotations. It proposes CMKT, which transfers appearance and action knowledge from cheaper cross-modal tasks for grounding without further training. Experiments on ActivityNet Captions and Charades-STA show effectiveness, including performance competitive with supervised works and outperforming existing unsupervised methods.
Problem
Unsupervised TSG must extract vision-language correlations without coarse video-query or fine-grained frame-query annotations, unlike annotation-dependent prior settings.
Method
CMKT collects object-guided appearance knowledge from Image-Noun and action knowledge from Video-Verb, synthesizes multi-action clips by copy-paste, and transfers both for inference without further training.
Results
CMKT is effective on ActivityNet Captions and Charades-STA, outperforming existing unsupervised methods and competitively beating supervised works.
Takeaways & Limitations
Cross-modal knowledge from cheaper annotated tasks can be directly generalized to guide unsupervised TSG without large-scale video-query pairs or frame-query pairs.
Takeaways & Limitations
The contexts of cheap multi-modal datasets are not always helpful for the grounding task, motivating more contextual and generalizable cross-modal transfer.
Abstract
from arXiv · showhide
This paper addresses the task of temporal sentence grounding (TSG). Although many respectable works have made decent achievements in this important topic, they severely rely on massive expensive video-query paired annotations, which require a tremendous amount of human effort to collect in real-world applications. To this end, in this paper, we target a more practical but challenging TSG setting: unsupervised temporal sentence grounding, where both paired video-query and segment boundary annotations are unavailable during the network training. Considering that some other cross-modal tasks provide many easily available yet cheap labels, we tend to collect and transfer their simple cross-modal alignment knowledge into our complex scenarios: 1) We first explore the entity-aware object-guided appearance knowledge from the paired Image-Noun task, and adapt them into each independent video frame; 2) Then, we extract the event-aware action representation from the paired Video-Verb task, and further refine the action representation into more practical but complicated real-world cases by a newly proposed copy-paste approach; 3) By modulating and transferring both appearance and action knowledge into our challenging unsupervised task, our model can directly utilize this general knowledge to correlate videos and queries, and accurately retrieve the relevant segment without training. Extensive experiments on two challenging datasets (ActivityNet Captions and Charades-STA) show our effectiveness, outperforming existing unsupervised methods and even competitively beating supervised works.
1 Introduction
The paper targets unsupervised temporal sentence grounding without video-query or frame-query annotations by transferring appearance and action knowledge from cheaper cross-modal tasks. Its CMKT network performs grounding without further training, and experiments on ActivityNet Captions and Charades-STA validate its effectiveness.
- Motivation: TSG locates a video segment whose activity semantically corresponds to a sentence query, but most prior methods require labor-intensive frame-query annotations.Weakly supervised methods reduce this reliance by using coarse video-query annotations, while unsupervised TSG excludes both annotation types.
- Motivation: Unsupervised TSG instead asks how to extract vision-language correlations without coarse video-query or fine-grained frame-query annotations.The paper proposes transferring knowledge from massively available, cheaply annotated image-noun and video-verb tasks.
- Method: CMKT transfers extracted appearance and action knowledge from other cross-modal tasks to search for the best-matching activity in unsupervised TSG.The network aggregates the transferred knowledge for inference without further training.
- Method: The appearance module learns object-guided matched region-noun information, while the action module combines single-action learning with copy-paste synthesis of multi-action videos.The copy-paste strategy adapts action representations to complex scenes containing multiple actions.
- Contributions: The paper makes a first attempt to transfer cross-modal knowledge into TSG and directly generalize pretrained appearance and action information without TSG annotations.This design eliminates complex training for precise grounding.
2 Related Works
Prior TSG work commonly uses fully supervised annotations and aligns predefined segment proposals with video-query information and annotated boundaries.
- Fully-Supervised TSG: Most existing TSG methods manually annotate all video-query pairs and precise segment boundaries under fully supervised settings.Top-down approaches may first define multiple segment proposals before aligning them with the query.
3 Our Method
CMKT addresses unsupervised TSG by collecting appearance knowledge from Image-Noun data and action knowledge from Video-Verb data, then refining and transferring both into TSG. Its action module combines single-action representations with copy-paste-generated multi-action representations to handle complex scenes.
- Problem definition: Unsupervised TSG removes correspondence and segment-boundary annotations between videos and queries.
- Appearance knowledge: CMKT collects region-noun appearance knowledge from Image-Noun datasets and averages relevant region features into noun-specific representations.
- Action knowledge: CMKT learns action knowledge from labeled Video-Verb clips using verb-specific consensus representations derived from action videos.
- Appearance knowledge: The appearance module targets region-aware foreground-object alignment with corresponding nouns rather than coarse whole-image associations.
- Action knowledge: Its multi-action branch copy-pastes clips between action-pure videos, with paste rate µ controlling the replaced-clip proportion, then combines pure and hybrid action features.
- Knowledge transfer: A knowledge-transfer module fine-tunes the collected appearance and action knowledge for TSG because source-task domain gaps and complicated scenes prevent direct use.
4 Experiment
Experiments evaluate CMKT on ActivityNet Captions and Charades-STA using IoU-based metrics and comparisons with fully supervised, weakly supervised, and unsupervised methods. CMKT remains competitive without supervision, while ablations support combining appearance, action, and transferred knowledge, including copy-paste augmentation.
- Datasets and Evaluation Metrics: ActivityNet Captions contains 20,000 YouTube videos and 100,000 language queries, with 17,031 video-query pairs used for testing.Queries average about 13.5 words, and videos average two minutes.
- Datasets and Evaluation Metrics: IoU=m measures the percentage of queries whose best result has IoU greater than m, while mIoU averages predicted and ground-truth temporal IoU.Evaluated thresholds are m ∈{0.3, 0.5, 0.7} for Charades-STA and m ∈{0.1, 0.3, 0.5, 0.7} for ActivityNet Captions.
- Comparison with State-of-the-Arts: Without supervision, CMKT remains competitive with weakly supervised and unsupervised methods on ActivityNet Captions and competitive with supervised methods on Charades-STA.The comparisons include fully supervised, weakly supervised, and unsupervised categories.
- Comparison with State-of-the-Arts: 3.40% and 3.12% improvements over DSCNet occur at IoU=0.3 and IoU=0.5 on ActivityNet Captions; Charades-STA gains are 3.65%, 2.23%, and 4.20% at IoU=0.3, IoU=0.5, and IoU=0.7.These comparisons are reported for the unsupervised setting.
- Ablation Study: CMKT(Full) outperforms ablations using only pre-trained features, only action knowledge, or only appearance information, supporting the combined module design.The reported ablations remove knowledge transfer, appearance knowledge, or action knowledge.
- Ablation Study: Average pooling outperforms min-pooling and max-pooling, while joint appearance and action knowledge further boosts performance and copy-paste synthesis improves multi-action grounding.Both single-action and multi-action features contribute to the action knowledge module.
- Qualitative Results: CMKT predicts more precise segment boundaries than fully supervised 2D-TAN and weakly supervised WSTAN in the visualized localization results.This comparison is reported from Figure 4.
5 Conclusion
The paper addresses unsupervised TSG through knowledge transfer from other cross-modal tasks, using general appearance and action knowledge without grounding training. CMKT is reported to outperform the existing unsupervised method by a large margin.
- Conclusion: CMKT transfers general knowledge from other cross-modal tasks to match videos and queries and retrieve target segments without further grounding training.The paper presents this as its knowledge-transfer perspective for unsupervised TSG.
- Conclusion: The paper identifies its transfer of knowledge from other multi-modal topics into TSG as a first attempt to do so.This novelty claim is stated by the authors.
- Conclusion: Experimental results validate CMKT’s effectiveness by outperforming the existing unsupervised method by a large margin.The conclusion summarizes the reported experimental outcome.
7 Limitations
The transferred cross-modal annotations are not always helpful for unsupervised temporal sentence grounding, motivating more contextual and generalizable transfer methods.
- Cross-modal datasets used for knowledge transfer may contain contexts that are not always helpful for the grounding task.The paper identifies more contextual and generalizable transfer across different topics as a promising future direction.