Vid-Morp: Video Moment Retrieval Pretraining from Unlabeled Videos in the Wild

1 INTRODUCTION

• Manual VMR annotations are costly, labor-intensive, difficult to scale, and can encode language and temporal biases.Prior unsupervised methods still require videos drawn from annotated datasets to be manually pre-cleaned, limiting scalability and leaving diverse in-the-wild videos less explored.Source: §1 · p. 1
• Vid-Morp offers more than 50K in-the-wild videos and 200K pseudo training samples, but the samples can contain idle videos, mismatched queries, and imprecise boundaries.The dataset is collected with minimal human involvement.Source: §1 · p. 1, Figure 1 · p. 2
• ReCorrect uses semantics-guided refinement to clean and initially adjust pseudo labels, then memory-consensus correction to progressively calibrate boundaries.The memory bank tracks model predictions and uses their consensus as a reference for boundary correction.Source: §1 · p. 1
• Zero-shot ReCorrect exceeds 75% and 80% of the best fully supervised performance on Charades-STA and ActivityNet Captions, respectively, while unsupervised ReCorrect reaches about 85% on both.The paper also reports zero-shot ReCorrect outperforming fully supervised methods tailored to two out-of-distribution benchmarks.Source: §1 · pp. 1–2

2 RELATED WORKS

• Fully supervised VMR methods rely on labor-intensive annotations, while existing unsupervised approaches use clean videos from manually annotated datasets and remain challenged by noisy real-world video.ProTeGe pretrains a feature-extraction backbone, whereas this work targets the retrieval model with a fixed backbone and supports zero-shot use without additional fine-tuning.Source: §2 · p. 2
• UniTVG's VMR performance reaches only half the metrics of other zero-shot approaches tailored to VMR, motivating datasets designed specifically for VMR.The comparison is reported in Table 2.Source: §2 · p. 2

3.1 Overview

• Vid-Morp addresses the reliance on clean videos from annotated datasets by using diverse, in-the-wild videos collected with minimal human intervention.Figure 2 illustrates video samples with sentence queries and temporal boundaries; the dark-blue box marks the described moment's boundary.Source: §3.1 · p. 2, Figure 2 · p. 3
• Vid-Morp contains over 50K in-the-wild videos and 200K training annotations, with five times ActivityNet Captions' number of videos and queries.Table 1 reports comparative training-split dataset statistics.Source: §3.1 · pp. 2–3, Table 1 · p. 3

3.2 Dataset Construction

• Dataset construction web-crawls videos for target activities, samples frames, and prompts a multimodal language model such as GPT-4o to generate descriptive sentences with frame indices mapped to timestamps.Human involvement is primarily limited to defining search keywords and designing the prompt.Source: §3.2 · p. 3

4 RECORRECT ALGORITHM

• ReCorrect tackles idle videos, unmatched video-query pairs, and inaccurate temporal boundaries with two phases: semantics-guided refinement and memory-consensus correction.The first phase removes erroneous samples and initially adjusts boundaries; the second uses a memory bank to correct boundaries by consensus.Source: §4 · p. 3

4.1.1 Semantics-Guided Refinement

• Frame-query semantic similarity guides sample cleaning and initial boundary adjustment, with moment contrastive scores used to filter training samples.The method removes the bottom R percent by score, then iteratively adjusts start and end times; T denotes the total number of video frames.Source: §4.1.1 · pp. 3–4

4.1.2 Memory Consensus Correction

• A memory bank stores candidate boundaries refined in the first phase, and the candidate with the highest consensus is selected as pseudo ground truth.Consensus is based on summed IoU with other memory instances; the model’s highest-confidence prediction is inserted into the bank.Source: §4.1.2 · pp. 4–5

4.2 Finetuning on Various Settings

• The pretrained model supports zero-shot inference, unsupervised finetuning with generated pseudo annotations and ReCorrect, and fully supervised finetuning with manual annotations.The fully supervised setting uses the same loss as SimBase.Source: §4.2 · p. 5

5.1 Datasets and Evaluation Metrics

• R@m is the percentage of language queries whose retrieved moment has IoU greater than threshold m.A retrieval counts as correct when its IoU with the ground-truth moment exceeds m.Source: §5.1 · p. 5
• The evaluation datasets are Charades-STA and ActivityNet Captions.The paper describes both as large-scale datasets.Source: §5.1 · p. 5
• IoU compares the retrieved temporal moment with the ground-truth moment.Source: §5.1 · p. 5

5.2 Implementation Details

• Pretrained CLIP extracts visual and textual features, and the model architecture matches SimBase.The video moment retrieval model uses the same architecture as the fully supervised SimBase model.Source: §5.2 · p. 5
• Semantics-guided refinement uses 256 frames, step size 5, cleaning ratio 40%, and α1 and α2 values of 0.22 and 0.92.Source: §5.2 · p. 5
• Training uses Adam, batch size 256, learning rate 0.0004, 15 pretraining epochs, and loss weight 0.7.Source: §5.2 · p. 5

5.3.3 Unsupervised Learning

• Unsupervised ReCorrect achieves approximately 85% of fully supervised SimBase’s overall performance on both datasets in mIoU.The table compares zero-shot, fully supervised, and unsupervised settings; its gray row expresses ReCorrect as a percentage of SimBase.Source: Table 2 · p. 6, §5.3.3 · p. 6

5.3.1 Zero-Shot Inference

• ReCorrect surpasses the previous best zero-shot model, VTG-GPT, by 7 points in R@0.7 and over 5 points in mIoU on both datasets.Direct pretraining on GPT4o’s original pseudo labels gives results similar to VTG-GPT and Lu et al.; ReCorrect consistently improves on that pretraining.Source: §5.3.1 · p. 5, Table 2 · p. 6

5.3.2 Fully-Supervised Learning

• ReCorrect improves GPT4o Pretraining by approximately 2 points at Charades R@0.7 and 0.5 points at ActivityNet R@0.3.Against SimBase, both methods improve R@0.5 by over 1.5 points on Charades and ActivityNet R@0.3 by around 1 point.Source: §5.3.2 · p. 5, Table 2 · p. 6

5.3.3 Unsupervised Learning

• Vid-Morp-pretrained versions show approximately 3- and 5-point gains over versions fine-tuned solely on Charades or ActivityNet.The comparison is between GPT4o Finetuning and ReCorrect Finetuning models with and without Vid-Morp pretraining.Source: §5.3.3 · p. 5, Table 2 · p. 6

5.3.4 Comparisons to Existing Pretraining Paradigms

• ReCorrect pretrains the VMR model itself, unlike ProTeGe’s backbone pretraining, enabling zero-shot inference; UniTVG has lower zero-shot scores than VMR-specific approaches.UniTVG achieves half of Luo et al.’s R@0.7 score and one-third of ReCorrect’s.Source: §5.3.4 · p. 6

5.3.5 Out-of-Distribution Scenarios

• ReCorrect beats SSL and VISA on Novel Composition and Novel Word, exceeds MomentDETR on Charades-CD, but ActivityNet-CD results are mixed.On ActivityNet CD, ReCorrect has a 5-point advantage in R@0.3 and a 2-point disadvantage in R@0.7. The authors report zero-shot ReCorrect requires neither fine-tuning nor OOD-specific algorithm design.Source: §5.3.5 · p. 6, Table 3 · p. 7, Table 4 · p. 7

5.3.6 Qualitative Comparisons

• Qualitative examples cover black-and-white and low-light video, animal and underwater activities, and compositional events requiring temporal reasoning.Source: §5.3.6 · p. 7

5.4.2 Impact of Cleaning Ratio

• Performance improves as cleaning rises from 0% to 30%, declines above 50%, and is satisfactory at a cleaning ratio of 20%–40%.The cleaning ratio is the percentage of data samples filtered out.Source: Figure 6 · p. 7, §5.4.2 · p. 8

5.4.1 Scability of Pretraining Dataset Size

• Doubling pretraining data yields linear performance gains from 6.3K to 12.7K, with continued but shallower gains from 25.3K to 200.3K.Performance is the average of R@m for m = 0.3, 0.5, and 0.7.Source: §5.4.1 · p. 7

5.4.3 Effectiveness of the Proposed Modules

• Removing any of label cleaning, boundary adjustment, or memory-consensus correction leads to notable declines in zero-shot and unsupervised performance.The paper reports that each module contributes positively in both settings.Source: Table 6 · p. 8, Table 5 · p. 8, §5.4.3 · p. 8

• Vid-Morp is a large-scale Video Moment Retrieval Pretraining dataset collected with minimal manual intervention.Source: §6 · p. 8
• ReCorrect addresses pseudo-label errors through semantics-guided refinement and memory-consensus correction.The first phase filters unpaired data and adjusts boundaries; the second uses a memory bank to track predictions and correct boundaries by consensus.Source: §6 · p. 9
• Experiments demonstrate ReCorrect’s effectiveness across various learning settings and strong generalizability.Source: §6 · p. 9