Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video

1 Introduction

• Earlier weakly supervised video grounding targets nouns or pronouns in single frames, where sparse language and single-frame appearance can leave similar objects ambiguous.The paper notes that a sentence describing an action and a tube showing temporal movement can better characterize the target.Source: §1 · pp. 1–2
• WSSTG localizes a sentence-matching spatio-temporal tube without fine-grained regional annotations during training.Unlike earlier weakly supervised video grounding, it targets natural sentences rather than only nouns or pronouns in a static frame.Source: §1 · pp. 1–2
• The proposed model uses multiple instance learning, an attentive interactor for fine-grained instance-sentence relationships, and ranking and diversity losses.The diversity loss strengthens matching for reliable pairs and penalizes unreliable ones during training.Source: §1 · p. 2
• VID-sentence adds natural sentence descriptions for 7,654 instances from 30 categories across 4,381 ImageNet VID videos.Annotators provide a sentence describing each extracted instance.Source: §1 · p. 2
• The authors report that extensive experiments show their method is superior to baseline approaches.The contribution statement does not specify the comparisons or values here.Source: §1 · p. 2

2 Related Work

• Fully supervised video grounding can require intensive regional annotation, especially for videos.Weakly supervised grounding instead uses image-sentence or video-sentence pairs.Source: §2 · p. 2
• The model architecture pairs an instance generator with an attentive interactor and trains them under multiple instance learning with ranking and diversity losses.The generator produces proposals; the interactor models their relationships with the sentence.Source: Figure 2 · p. 3
• Earlier weakly supervised video methods ground nouns or pronouns in specific frames, while this work maps natural sentences to spatio-temporal tubes.The comparison is stated explicitly for Zhou et al. (2018).Source: §2 · p. 3

3 Method

• The model uses an instance generator and attentive interactor, trained with ranking and diversity losses, to identify a tube matching a sentence without regional training annotations.The ranking loss distinguishes aligned from unaligned video–sentence pairs; the diversity loss strengthens reliable proposal matches and penalizes unreliable ones.Source: §3 · p. 3, §3.3 · p. 5

3.1 Instance Extraction

• A Faster R-CNN region proposal network detects frame-level boxes, which are linked into tubes using box confidence and adjacent-frame IoU.The method repeatedly extracts the highest-energy tube with the Viterbi algorithm and removes its boxes before finding the next proposal.Source: §3.1 · p. 3
• Each tube is represented by concatenated I3D RGB, I3D flow, and RoI-pooled features averaged over 20 temporal segments.The authors use segment averaging because a proposal may contain thousands of boxes.Source: §3.1 · p. 4

3.2 Attentive Interactor

• The attentive interactor couples interaction with matching-behavior characterization to score instance-proposal and sentence pairs.Figure 3 labels these two components; the interactor uses their relationships to yield matching scores.Source: §3 · p. 3, §3.2 · p. 4, Figure 3 · p. 4

3.2.1 Interaction

• LSTMs encode the visual and sentence sequences, and each visual feature attends over sentence features to form visually guided sentence representations.This interaction exploits fine-grained relationships between visual and sentence representations.Source: §3.2.1 · pp. 4–5

3.2.2 Matching Behavior Characterization

• Matching behavior is computed between visual features and their visually guided sentence features, using cosine similarity in this paper.The proposal-level score is the average of the segment-level matching scores.Source: §3.2.2 · p. 5

3.3 Training

• Training combines ranking of aligned versus unaligned video–sentence pairs with an entropy-based diversity loss that strengthens reliable matches and penalizes unreliable ones.The diversity loss encourages a diverse distribution of proposal matching scores within an aligned video.Source: §3.3 · p. 5

3.4 Inference

• At inference, the proposal with the strongest matching behavior for the query sentence is selected as the grounding result.The method extracts candidate proposals from the test video and scores each with the attentive interactor.Source: §3.4 · p. 5

VID-sentence Annotation

• VID was chosen for its diverse categories and complex scenarios, while its dense boxes and instance IDs reduce annotation work for validation and testing.The Person-sentence dataset contains only people, limiting its suitability for WSSTG.Source: §4 · p. 6
• The dataset is built by trimming VID training and validation videos into instances and collecting one sentence description for each retained instance.Instances judged too difficult to describe uniquely and precisely are discarded; 7,654 videos with descriptions remain.Source: §4 · p. 6

Dataset Statistics

• VID-sentence contains 6,582/536/536 described spatio-temporal instances for training/validation/testing and covers all 30 VID categories.Its vocabulary has 1,823 words, and descriptions average 13.2 words.Source: §4 · p. 6

VID-sentence Annotation

• VID regional annotations are not used in the experiments, preserving the paper’s weakly-supervised training setting.The VID test set is omitted because its spatio-temporal annotations are unavailable.Source: §4 · p. 6

5 Experiments

• The experiments compare baselines on VID-sentence, conduct an ablation study, and test generalization on Person-sentence.Source: §5 · p. 6

5.1 Experimental Settings

• Localization is accurate when average overlap across annotated frames exceeds η; evaluations use thresholds 0.4, 0.5, and 0.6.Source: §5.1 · p. 7

5.2 Performance Comparisons

• Our model achieves the best results among methods on VID-sentence.Source: §5.2 · p. 7, Table 2 · p. 7
• Our method finds a more accurate proposal than GroundeR+LSTM and DVSA+LSTM, with greater temporal consistency than the frame-level variant.Source: §5.2 · p. 7, Figure 5 · p. 8

5.3 Ablation Study

• Both components improve over Base; attention favors matching words, while diversity penalizes lower-IoU proposals and strengthens higher-IoU ones.Source: §5.3 · p. 8, Figure 6 · p. 8, Table 3 · p. 8

5.4 Experiments on Person-sentence Dataset

• On Person-sentence, the attentive interactor without diversity loss outperforms all baselines, and diversity loss further improves performance.Source: Table 4 · p. 8, §5.4 · p. 9

• The task maps a sentence and video to a semantically matching spatio-temporal tube without spatio-temporal annotations during training.Source: §6 · p. 9
• The model uses an attentive interactor and diversity loss to learn relationships between instance proposals and the sentence within multiple instance learning.Source: §6 · p. 9
• The authors report effective experimental results and introduce VID-sentence as a benchmark for the task.Source: §6 · p. 9

Supplementary Material

• Visual instance features use averaging, NetVLAD, or an LSTM; averaging pools segments, while the LSTM uses its last hidden state.Source: §7 · p. 12
• The modified Zhou et al. baseline replaces noun encoding with an LSTM for natural sentences and removes frame-wise loss weighting.Source: §7 · p. 12
• Faster-RCNN supplies 30 region proposals per frame and RoI features; I3D supplies RGB and flow features extracted from video segments.Source: §7 · p. 12