PaperlayerView exact revision on arXiv

Source-linked AI summary

TF-PRVR: Training-Free Partially Relevant Video Retrieval

Giyeol Kim, Chanho Eom

arXiv:2610.07925v1cs.CV

TL;DR

Existing PRVR methods face fixed video decomposition and source-domain overfitting, limiting how well their representations fit varied untrimmed videos and unseen datasets. TF-PRVR uses frozen vision-language features to build adaptive hierarchical segments and propagate relevance across a multi-scale graph. Experiments report consistent training-free retrieval across datasets with diverse visual and temporal characteristics.

  • Problem

    Existing PRVR methods use fixed video decomposition that can dilute semantics and task-specific training associated with source-domain overfitting.

    Source: §1 · p. 2, §3 · p. 3

  • Method

    TF-PRVR uses frozen vision-language features to form adaptive hierarchical segments, then propagates query relevance across a unified multi-scale graph and scores temporally aligned relevance.

    Source: §1 · p. 2, §4.1 · p. 5, §4.3 · p. 7

  • Results

    TF-PRVR demonstrates consistent training-free retrieval performance across datasets with diverse visual and temporal characteristics.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The experiments support training-free PRVR as a practical direction across datasets with diverse visual and temporal characteristics.

    Source: §1 · p. 2

    Performance depends on the frozen backbone’s zero-shot capability and may be limited in specialized domains; precomputing and storing the graph and propagation kernel also increases indexed query latency and memory.

    Source: §6 · p. 10

Abstract

from arXiv · show

Partially Relevant Video Retrieval (PRVR) aims to retrieve untrimmed videos containing moments relevant to a given text query. Despite recent progress, existing PRVR methods suffer from two key limitations: a fixed video decomposition scheme that causes semantic dilution, and source-domain overfitting induced by task-specific training. In this paper, we propose TF-PRVR, the first training-free framework for PRVR. TF-PRVR leverages frozen vision-language features to construct video-specific hierarchical representations. It derives temporal semantic signals from frame-level features and applies frequency-based multi-scale analysis to identify adaptive temporal boundaries, producing hierarchical segments with coherent event-level semantics. Built on these segments, TF-PRVR constructs a unified multi-scale graph and propagates query relevance across temporally and semantically related nodes. A moment-aware scoring strategy then aggregates temporally aligned relevance across scales, emphasizing consistently supported moments while suppressing isolated false responses. Without task-specific training, TF-PRVR preserves the general-purpose alignment capability of pre-trained vision-language models and avoids dataset-specific overfitting. Extensive experiments demonstrate consistent performance across datasets with diverse visual and temporal characteristics, suggesting a practical direction for training-free PRVR.

1 Introduction

PRVR addresses retrieval from untrimmed videos, where only some moments match a text query. TF-PRVR avoids task-specific training and fixed clip decomposition by building adaptive, multi-scale representations and propagating relevance across them.

1 Introduction

  • Untrimmed videos contain multiple distinct events, so conventional text-to-video retrieval’s fully relevant, pre-trimmed-video assumption does not fit the PRVR setting.PRVR retrieves videos when specific segments, rather than the entire video, correspond to the query.Source: §1 · p. 1
  • Fixed-size clip aggregation can merge frames from distinct events, diluting semantics and making query-relevant moments less discriminable.The paper gives kitesurfing and volleyball frames pooled together as an example of an ambiguous clip representation.Source: §1 · p. 2
  • Training-based PRVR methods can lose retrieval accuracy on unseen datasets; MSC-PRVR’s cross-domain R@1 falls below zero-shot CLIP’s 16.2%.MSC-PRVR reaches 5.7% and 4.9% R@1 when trained on Charades-STA and ActivityNet Captions, respectively, versus 16.2% for zero-shot CLIP.Source: §1 · p. 2
  • TF-PRVR uses frozen vision-language features, frequency-based adaptive boundaries, a unified multi-scale graph, and moment-aware scoring without PRVR-specific optimization.Its scoring aggregates temporally aligned relevance across scales to emphasize consistently supported moments and suppress isolated responses.Source: §1 · p. 2

2 Related Work

Prior PRVR work models untrimmed videos at frame and clip scales using several strategies for constructing or compressing clip representations. TF-PRVR introduces frequency-based modeling of semantic transitions in frozen vision-language features to create adaptive temporal boundaries.

Partially Relevant Video Retrieval

  • PRVR treats a video as relevant when it contains at least one query-matching moment; prior methods use sliding windows, implicit Gaussian modeling, prototypes, or hyperbolic learning.The reviewed approaches balance context modeling, retrieval effectiveness, and efficiency in different ways.Source: §2 · p. 3
  • Existing PRVR methods still rely on fixed video decomposition and task-specific training, leaving them vulnerable to semantic dilution and domain overfitting.The paper contrasts this with video-specific hierarchical representations and a training-free retrieval pipeline.Source: §2 · p. 3

Frequency-Based Temporal Modeling

  • TF-PRVR applies frequency-based temporal modeling to semantic transitions in frozen vision-language features, rather than low-level motion or pixel signals, to identify adaptive boundaries without moment-level annotations.The related work situates this approach alongside Fourier- and wavelet-based temporal modeling used in other video tasks.Source: §2 · p. 3

3 Revisiting the Limitations of Existing PRVR Methods

The analyses examine source-domain overfitting and rigid video decomposition as limitations of existing PRVR methods. They relate transfer differences to dataset-specific score distributions and show that fixed clips, fixed branch weights, and independent pooling do not consistently fit video temporal structure.

3.1 Analysis of Source-Domain Overfitting

  • Task-specific training is associated with larger cross-dataset feature-distribution discrepancies than frozen CLIP features, consistent with degraded retrieval on unseen datasets.For example, cross-dataset MMD is 0.002 for frozen CLIP-L features between TVR and Charades-STA, compared with 0.054 for HLFormer and 0.063 for MSC-PRVR.Source: §3.1 · p. 4

3.2 Analysis of Fixed Video Decomposition

  • Fixed clips span different temporal ranges across datasets and interact differently with moment duration, producing opposite short- versus long-video performance trends on ActivityNet Captions and TVR.On ActivityNet Captions, long videos perform better for HLFormer and GMMFormer; on TVR, short videos perform better for both methods.Source: §3.2 · p. 4
  • For GMMFormer, the optimal clip/frame weighting differs by dataset, and query-wise optimal weights vary substantially, suggesting one global weight may be suboptimal.The reported dataset-specific optima are 0.6 for ActivityNet Captions and 0.8 for Charades-STA.Source: §3.2 · p. 5
  • Independent max-pooling can select clip and frame representations from different temporal regions; same-group rates are 28.8% for ActivityNet Captions, 48.6% for Charades-STA, and 38.5% for TVR.The normalized-distance distribution is long-tailed, indicating severe misalignment for a non-negligible number of queries.Source: §3.2 · p. 5

4 TF-PRVR: Training-Free Partially Relevant Video Retrieval Framework

TF-PRVR uses frozen vision-language representations to build video-specific hierarchical segments, then propagates query relevance over a unified multi-scale graph. Its final score favors temporal moments supported consistently across scales.

4.1 Overview

  • TF-PRVR extracts frame and query features, adaptively segments each video, propagates query-segment relevance over a multi-scale graph, and scores the video by moment-aware aggregation.It avoids task-specific training and fixed clip counts.Source: §4.1 · p. 5

4.2 Video-Specific Hierarchical Segmentation

  • Frame-feature velocity changes define a temporal signal, with larger cosine dissimilarities between consecutive velocities indicating potential event boundaries.The signal measures changes in the direction of semantic evolution.Source: §4.2 · p. 5
  • A discrete wavelet transform decomposes the signal into scales, whose detail coefficients yield boundary responses and video-specific multi-scale boundary sets.Coarser scales capture long-range transitions, while finer scales capture short-range changes; the number of decomposition levels varies by video.Source: §4.2 · pp. 5–6
  • Detected boundaries partition videos into contiguous segments represented by average-pooled frame features, preserving coherent visual semantics and reducing semantic dilution.The number of segments varies across videos and scales.Source: §4.2 · p. 6

4.3 Graph-Based Retrieval with Multi-Scale Consistency

  • A unified graph links segments within scales by visual similarity and temporal proximity, and across adjacent scales by temporal overlap and semantic similarity.The graph combines intra- and inter-level affinities into a normalized adjacency matrix for relevance propagation.Source: §4.3 · pp. 6–7
  • Closed-form propagation refines query-segment relevance, and the final video score takes the strongest temporal consensus across scales, suppressing isolated single-scale peaks.At each time point, TF-PRVR averages propagated relevance from the segments covering that point.Source: §4.3 · p. 7

5 Experiments

Experiments show TF-PRVR performs competitively without PRVR-specific training, including under cross-dataset transfer. Ablations and qualitative results support its video-specific hierarchy and multi-scale relevance design.

5.1 Experimental Setup

  • The evaluation uses ActivityNet Captions, Charades-STA, and TVR, reporting R@K for K ∈ {1, 5, 10, 100} and their sum, SumR.The implementation uses frozen CLIP-L/14, SigLIP-L/14, or EVA-CLIP-L/14 features, with db4 wavelets and propagation coefficient β = 0.7.Source: §5.1 · p. 7

5.2 Comparison with the State-of-the-Art

  • TF-PRVR‡ scores 200.8 SumR on ActivityNet Captions without source-domain PRVR training, exceeding all source-trained baselines evaluated on that target.MSC-PRVR scores 28.1, 123.8, and 106.3 SumR when trained on Charades-STA, TVR, and their combination, versus 202.1 with in-domain ActivityNet training.Source: Table 5 · p. 8, §5.2 · p. 8

5.3 Ablation Study

  • Across frozen backbones, TF-PRVR raises SumR from 172.9 to 193.9 with CLIP-L/14, 178.7 to 198.5 with SigLIP-L/14, and 180.4 to 200.8 with EVA-CLIP-L/14.The modules operate on frozen frame and text features without task-specific optimization, enabling plug-and-play use across backbones.Source: §5.3 · p. 8

5.2 Comparison with the State-of-the-Art

  • TF-PRVR improves over CLIP-L zero-shot SumR from 172.9 to 193.9 on ActivityNet Captions, 73.1 to 80.8 on Charades-STA, and 167.2 to 178.3 on TVR.It reaches R@1 17.5 on ActivityNet Captions, comparable to MSC-PRVR's 17.7, while training-based methods achieve higher overall in-domain performance.Source: §5.2 · p. 8, Table 6 · p. 9

5.3 Ablation Study

  • On ActivityNet Captions, hierarchical segmentation, graph propagation, and moment-aware scoring successively raise SumR from 186.6 to 194.7, 197.8, and 200.8 after the fixed-segmentation variant.The ablation attributes reduced semantic dilution to hierarchical segmentation and improved multi-scale temporal consistency to graph propagation and moment-aware scoring.Source: Table 7 · p. 9, §5.3 · p. 9
  • Velocity-direction change achieves the best temporal-signal SumR of 200.8, outperforming other variants by 3.2–9.0 points.It captures changes in semantic-evolution direction; the alternatives are uniform fixed intervals, adjacent-feature dissimilarity, and windowed self-similarity.Source: §5.3 · p. 9

5.4 Qualitative Result

  • Figure 5 shows finer hierarchy levels isolating local events and coarser levels capturing broader context, with example queries matching segments across levels.For the 107-frame video, fixed clips would average about three frames and could fragment relevant moments or mix them with unrelated content.Source: Figure 5 · p. 10, §5.4 · p. 10

6 Limitations and Future Work

TF-PRVR's effectiveness is bounded by its frozen backbone and the storage and latency costs of query-independent graph components.

6 Limitations and Future Work

  • Performance depends on the frozen backbone's zero-shot capability and may be limited in specialized domains; precomputing and storing the multi-scale graph and propagation kernel also raises indexed-query latency and memory.These components are query-independent and computed once for reuse across queries.Source: §6 · p. 10

7 Conclusion

TF-PRVR combines training-free video-specific hierarchies with multi-scale graph propagation and moment-aware scoring. Experiments support its effectiveness across diverse datasets without source- or target-specific PRVR training.

7 Conclusion

  • TF-PRVR addresses source-domain overfitting and rigid video decomposition by using frozen vision-language alignment, adaptive temporal boundaries, and a query-independent multi-scale graph with relevance propagation and moment-aware scoring.The conclusion presents it as a practical direction for training-free PRVR.Source: §7 · p. 10

Appendix

  • Table A1 compares methods on additional video datasets and identifies the PRVR training data; TF-PRVR and CLIP zero-shot require no PRVR-specific training.Source: Table A1 · p. 14

A.1 Evaluation on Additional Diverse Video Datasets

On four additional datasets, TF-PRVR improves over zero-shot and source-trained PRVR baselines without PRVR-specific training.

A.1 Evaluation on Additional Diverse Video Datasets

  • TF-PRVR improves SumR over the zero-shot baseline by 20.9 on QVHighlights, 15.1 on EPIC-Kitchens-100, 23.4 on TACoS, and 15.2 on YouCook2, while outperforming source-trained baselines on all four.The datasets span consumer videos, egocentric kitchen activities, fine-grained cooking actions, and procedural instructional videos.Source: §A.1 · p. 14

A.2 Additional Cross-Dataset Transfer Results

Additional transfer results show substantial cross-dataset degradation for trained PRVR models, while TF-PRVR exceeds transfer-trained baselines on Charades-STA and TVR without source-domain PRVR training.

A.2 Additional Cross-Dataset Transfer Results

  • On Charades-STA, TF-PRVR reaches 85.4 SumR without source-domain training, exceeding transfer-trained baselines; MSC-PRVR falls from 99.7 in-domain to 47.7 and 47.4 under transfer.The comparisons cover in-domain, cross-dataset transfer, and training-free settings.Source: §A.2 · p. 14, Table A2 · p. 15
  • On TVR, TF-PRVR reaches 183.6 SumR without source-domain PRVR training, exceeding transfer baselines but not MSC-PRVR’s 263.1 in-domain result.MSC-PRVR records 77.4 and 25.6 SumR when trained on ActivityNet Captions and Charades-STA, respectively, and 73.7 when trained on both.Source: §A.2 · p. 14, Table A3 · p. 15

B.1 Direct Analysis of Segmentation Quality

The analysis finds that video-specific segmentation better aligns and compacts relevant moments than fixed decomposition; budget-matched tests favor its more informative temporal boundaries.

B.1 Direct Analysis of Segmentation Quality

  • The post-hoc GT audit uses the best-tIoU segment per query to assess alignment, coverage, fragmentation, irrelevant-frame inclusion, and semantic coherence; GT moments are evaluation-only.Semantic coherence is measured by intra-segment dispersion of frozen vision-language features.Source: §B.1 · pp. 15–16
  • Across evaluated datasets, video-specific segmentation improves temporal alignment and GT coverage while reducing fragmentation, irrelevant-frame inclusion, and feature dispersion versus fixed decomposition.Table A4 reports these measures, with higher values preferred for alignment, R@0.5, and coverage, and lower values preferred for the other measures.Source: §B.1 · p. 16, Table A4 · p. 16

B.2 Budget-Matched Boundary Comparison

  • With 160 candidates per video, velocity-direction boundaries outperform uniform intervals, adjacent-feature dissimilarity, and self-similarity in SumR, approaching unconstrained adaptive segmentation.The unconstrained setting adapts candidate counts to video temporal structure.Source: §B.2 · p. 16, Table A5 · p. 16

B.3 Comparison with Training-Free Baselines

Using the same frozen features, adaptive segmentation improves training-free retrieval over fixed segmentation, and graph propagation and moment-aware scoring add further gains.

B.3 Comparison with Training-Free Baselines

  • Under the same GP and MAS pipeline, adaptive segmentation improves over fixed segmentation by 9.0, 4.9, 7.3, and 11.5 SumR on ActivityNet Captions, Charades-STA, TVR, and QVHighlights, respectively.The comparison uses training-free baselines with the same frozen features.Source: §B.3 · p. 17
  • Adaptive segmentation alone improves over fixed segmentation by 5.2, 2.7, 2.8, and 6.6 SumR on ActivityNet Captions, Charades-STA, TVR, and QVHighlights, respectively.Source: §B.3 · p. 17
  • Among fixed temporal aggregation baselines, exhaustive multi-scale sliding windows perform best; all fixed aggregation variants improve over frame-level zero-shot retrieval.Graph propagation and moment-aware scoring provide additional gains.Source: §B.3 · p. 17

B.4 DWT Wavelet Basis

This section summarizes TF-PRVR’s stability and ablation evidence, alongside efficiency, limitations, related-task distinctions, and implementation details. It also reports evaluation and research-practice disclosures, including the absence of error bars and significance tests.

B.4 DWT Wavelet Basis; B.5 Number of DWT Decomposition Levels

  • db4 and automatic DWT level selection each achieve the best SumR of 200.8 and are adopted as defaults.For the wavelet basis, Haar, db8, and sym4 score 197.7, 200.6, and 196.4; db4 balances localized transition detection with stability. For decomposition depth, L = 2, 3, and 4 score 195.6, 199.7, and 200.3, respectively; automatic selection adapts depth to each video.Source: §B.4 · p. 17, §B.5 · p. 18

B.8 Performance Comparison Across Video Length

  • Short- and long-video SumR scores are 193.7 and 194.0 on ActivityNet Captions, 80.7 and 80.8 on Charades-STA, and 178.4 and 178.1 on TVR.The authors attribute this stability to video-specific hierarchical segmentation from semantic transition boundaries rather than fixed intervals, whose clip spans vary with video duration.Source: Table A8 · p. 18, §B.8 · p. 18

B.6 Graph Components

  • The full graph achieves SumR 200.8, outperforming visual-only edges at 198.4, temporal-only edges at 188.9, intra-level-only at 195.2, and inter-level-only at 194.6.The results indicate that semantic similarity is more informative than temporal proximity alone, while both within-scale and cross-scale connections contribute complementary cues.Source: §B.6 · p. 18

B.7 Propagation Strength β

  • Propagation strength β = 0.7 achieves the best SumR of 200.8; β = 0.1, 0.3, and 0.9 score 197.4, 199.6, and 198.6, respectively.Small β underuses graph propagation, whereas β = 0.9 can oversmooth query-specific relevance; β = 0.7 is the default across datasets.Source: §B.7 · p. 18

B.9 Failure Case

  • Low-variance signals from recurring indoor backgrounds can make coarser DWT levels merge distinct moments, while finer segmentation and multi-scale graph consistency mitigate these errors.The cited example includes recurring background frames #2, #16, #30, and #85.Source: §B.9 · p. 18

B.10 Computational Efficiency

  • On ActivityNet Captions, TF-PRVR takes 49.08 s end to end versus MSC-PRVR’s 48.36 s, but uses 17.24 ms query latency and 3.80 GB indexed memory versus 9.78 ms and 0.48 GB.The end-to-end difference is 0.72 s (1.5%); query-independent hierarchy, graph, and propagation-kernel construction can be amortized across later queries, making index and query-time memory the main deployment trade-off.Source: Table A9 · p. 19, §B.10 · p. 19

C Comparison with Video Moment Retrieval and Temporal Grounding; D Positive and Negative Societal Impacts

  • Unlike VMR and VCMR, PRVR ranks untrimmed videos for query relevance without requiring moment-level supervision or boundary outputs, so localization methods are not naturally aligned with its protocol.VMR localizes a relevant moment within a given video, while VCMR retrieves a video and temporal moment from a corpus; precise moment annotations are labor-intensive, making PRVR a more scalable search formulation. Related work includes VMR [1] [26], VCMR [23] [48] [20], zero-shot VMR [29], and multi-scale or graph-based grounding [37]. TF-PRVR may aid information access and reduce task-specific annotation costs, but risks include privacy-invasive use and inherited backbone biases.Source: C Comparison with Video Moment… · p. 19, D Positive and Negative Societal Impacts · p. 19

E Algorithm Details; NeurIPS Paper Checklist

  • TF-PRVR’s training-free pipeline has video-specific hierarchical segmentation and graph-based retrieval stages, with graph structures and propagation kernels precomputed independently of queries.The paper says experiments specify relevant hyperparameters and standard dataset splits, provides source code, and reports that training details such as optimizers and learning rates do not apply.Source: E Algorithm Details · p. 20, NeurIPS Paper Checklist · pp. 23–24

NeurIPS Paper Checklist

  • The paper reports standard retrieval metrics on fixed benchmark splits but does not provide error bars or other statistical-significance information.The checklist cites adherence to prior PRVR evaluation protocols as its justification.Source: NeurIPS Paper Checklist · p. 24
  • The paper says it provides the computer resources used for experiments, including the GPU and CPU environment.Source: NeurIPS Paper Checklist · p. 25
  • The authors confirm compliance with the NeurIPS Code of Ethics and say privacy, fairness, societal concerns, and societal impacts are discussed in supplementary material.The checklist separately states that societal impacts are included in the supplementary material.Source: NeurIPS Paper Checklist · p. 25
  • The work does not release a new high-risk model, generative model, scraped dataset, or newly collected sensitive data, and introduces no new assets.The paper also says its datasets and pretrained models are cited in the references.Source: NeurIPS Paper Checklist · pp. 26–27
  • Experiments use existing publicly available datasets and involve no crowdsourcing or human subjects.Source: NeurIPS Paper Checklist · p. 27
  • LLMs were used only for grammar checking and editing, not as an important or original component of the core method.Source: NeurIPS Paper Checklist · p. 28

Related papers

Loading 2610.07925v1…