Source-linked AI summary
Intrinsic Temporal Adaptation of CLIP for Partially Relevant Video Retrieval
Hyun Seok Seong, Woojin Jun, SuBeen Lee, Jae-Pil Heo
TL;DR
PRVR must retrieve videos containing query-relevant moments despite limited temporal coverage, but independently encoded CLIP frames can yield imprecise evidence. ITA adds temporal interaction inside CLIP and propagates supervision across affinity-selected frames, achieving state-of-the-art benchmark performance and stronger temporal grounding.
Problem
PRVR requires fine-grained evidence for query-relevant moments, while standard CLIP encodes frames independently without explicit temporal context.
Method
ITA adapts the last few CLIP visual layers to attend over neighboring frames and uses Affinity-Weighted Gradient Propagation to supervise multiple high-affinity frames while freezing the backbone.
Results
ITA achieves state-of-the-art performance on four PRVR benchmarks, with fewer trainable parameters, lower video-encoder computation, stronger cross-dataset generalization, and more temporally grounded frame representations.
Takeaways & Limitations
Temporal interaction within CLIP and multi-frame supervision improve PRVR retrieval and the temporal precision of frame-level evidence.
Takeaways & Limitations
Because PRVR supplies only video-level supervision, CLIP representation biases may reinforce frames that align strongly with CLIP but are not the most temporally accurate evidence.
Abstract
from arXiv · showhide
Partially Relevant Video Retrieval (PRVR) aims to retrieve untrimmed videos that contain moments relevant to a text query. Since the target moment occupies only a portion of the video, PRVR requires retrieval based on fine-grained understanding beyond coarse video-level matching. However, existing methods often rely on frozen CLIP frame features, which lack temporal understanding. Even with recent progress in parameter-efficient CLIP adaptation, video-level predictions can still be supported by imprecise frame-level evidence. In this paper, we propose an Intrinsic Temporal Adaptation (ITA) framework for PRVR. First, our Backbone-Internal Temporal Adaptation allows the last few visual transformer layers to attend over groups of neighboring frames. This provides temporally aware frame embeddings while keeping CLIP frozen and training only adaptation parameters. Second, we introduce Affinity-Weighted Gradient Propagation to address the weakly supervised nature of PRVR, softly aggregating top-$k$ frames based on text-frame affinities and propagating learning signals to multiple query-relevant frames. Our method achieves state-of-the-art performance on PRVR benchmarks, demonstrates robust cross-dataset transfer, and retrieves substantially more accurate frame-level evidence within ground-truth query-relevant moments. Our code is available at github.com/hynnsk/ITA.
1 Introduction
PRVR retrieves videos whose query-relevant content may occupy only a temporal portion, requiring fine-grained evidence rather than coarse video-level semantics. ITA introduces temporal adaptation inside CLIP and affinity-weighted supervision to improve temporal grounding and retrieval performance efficiently.
- PRVR requires identifying query-supporting moments within videos instead of relying only on coarse video-level semantics.
- Frozen CLIP encodes frames independently without explicit temporal context, leaving temporal modules to derive moment-level semantics from isolated features.
- ProPy can produce imprecise frame-level evidence: its transition patterns recur across unrelated videos, and strict top-k retrieval degrades rapidly as k increases.
- Backbone-Internal Temporal Adaptation lets the last few visual-encoder layers attend over neighboring frames, conditioning each frame representation on local temporal context while keeping CLIP frozen.
- Affinity-Weighted Gradient Propagation softly aggregates top-k frames by text-frame affinity and propagates learning signals to multiple likely relevant frames under video-level supervision.
- ITA achieves strong PRVR retrieval performance with lightweight CLIP adaptation, while reporting state-of-the-art performance, backbone generality, cross-dataset transfer, and temporally precise frame-level retrieval.
2 Related Work
Related work distinguishes conventional global text-to-video retrieval from PRVR, where only a temporal portion may match the query. The proposed framework addresses this setting by adapting CLIP internally and propagating supervision across multiple relevant frames.
- 2 Related Work: Traditional text-to-video retrieval generally assumes global video-text relevance, which may fail when a query matches only a short temporal portion.
- 2 Related Work: PRVR methods identify partially relevant regions through event alignment, Gaussian-based representations, span anchors, and other temporal modeling strategies.
- 2 Related Work: The proposed framework performs temporal attention over grouped neighboring frames inside CLIP and propagates learning signals to multiple query-relevant frames.
- 2 Related Work: Recent work also improves weakly supervised representation learning through knowledge distillation, prototype-based clip representations, ambiguity-aware objectives, semantic-collapse mitigation, and hierarchical partial relevance.
3 Method
ITA adapts CLIP internally so frame embeddings incorporate temporal context, then uses affinity-weighted top-k aggregation to distribute weakly supervised learning across likely relevant frames.
- Preliminary: PRVR training receives video-level labels without annotations identifying the query-relevant frame or temporal segment.
- Backbone-Internal Temporal Adaptation: Backbone-Internal Temporal Adaptation lets neighboring frames interact in the last visual layers before producing temporally contextualized frame embeddings.Grouped temporal attention processes concatenated tokens from consecutive frames and is applied mainly in deeper layers suited to temporal event understanding.
- Backbone-Internal Temporal Adaptation: Learnable temporal position signals and relative temporal biases make grouped attention sensitive to frame order and relative offsets.The relative bias allows attention to distinguish tokens by both visual content and temporal position.
- Parameter-Efficient Adaptation: The adaptation freezes pretrained CLIP parameters and trains lightweight LoRA and temporal parameters for parameter-efficient adaptation.The text and visual encoders receive LoRA on attention input projections, while other backbone projections, MLPs, and normalization parameters remain frozen.
- Affinity-Weighted Gradient Propagation: Affinity-Weighted Gradient Propagation selects top-k frames by text-frame affinity, softly weights them, and forms the video representation from their weighted sum.Using k > 1 distributes the contrastive gradient across multiple high-affinity frames rather than primarily a single frame.
- Affinity-Weighted Gradient Propagation: AWGP remains query-adaptive because each query-video pair selects its own candidates and affinity weights can reduce weak candidates’ contributions.Inference uses the same top-k affinity-weighted scoring procedure, with negligible additional cost compared with top-1 aggregation.
4 Experiment
Experiments across four PRVR benchmarks show that ITA achieves strong retrieval with efficient adaptation, while ablations clarify the roles of temporal interaction, affinity-weighted supervision, and selection strategy.
- Experiment Settings: The evaluation covers TVR, ActivityNet Captions, Charades-STA, and QVHighlights, using recall-based R@K metrics and their sum, SumR.The implementation samples 32 frames per video and uses CLIP-B/32 with shared hyperparameters across datasets.
- Comparison with the State of the Art: ITA achieves the best overall performance across the evaluated benchmarks, improving SumR over ProPy by 21.4 points on ActivityNet Captions.It also achieves state-of-the-art performance on Charades-STA and QVHighlights, with a substantial gain on TVR.
- Comparison with the State of the Art: ITA consistently improves over ProPy across CLIP and video-language backbones, including CLIP ViT-B/16, CLIP4Clip-B/32, and InternVideo-MM-B/16.The average SumR comparison spans four PRVR benchmarks.
- Ablation Study: Adding Grouped Temporal Attention, Temporal Bias, and Affinity-Weighted Gradient Propagation each improves the parameter-efficient baseline, while their combination performs best.The ablation attributes complementary benefits to temporal representation learning and affinity-weighted training.
- Ablation Study: Selecting multiple high-affinity frames outperforms top-1 aggregation, but excessive selection dilutes supervision; the default is k=4 and τ=0.05.Very small temperatures approach top-1 weighting, whereas very large temperatures make weights nearly uniform and may emphasize less relevant frames.
- Ablation Study: Temporal grouping improves frame-independent encoding, but overly broad interaction can weaken frame specificity; sliding grouping raises average SumR to 199.5 at 313.6 GFLOPs.The retained non-overlapping configuration balances accuracy and computation, while overlapping grouping reaches 198.2 average SumR.
- Ablation Study: No fixed affinity threshold consistently beats top-k selection across datasets because absolute similarities are not equally calibrated, whereas relative ranking is more stable.Affinity-Weighted Gradient Propagation still assigns query-adaptive weights to the selected frames rather than treating them equally.
- Generalization and Frame-Level Retrieval: Cross-dataset evaluation trains only on QVHighlights and tests on other PRVR benchmarks, while frame-level retrieval evaluates whether retrieved frames lie inside annotated relevant moments.ITA generalizes more robustly than existing methods and consistently outperforms ProPy on frame-level retrieval across datasets and recall levels.
5 Conclusion
The paper concludes that ITA improves PRVR by adapting CLIP internally to temporal context and spreading learning signals across relevant frames. Experiments support stronger performance, efficiency, temporal grounding, and cross-dataset generalization.
- Backbone-Internal Temporal Adaptation injects temporal interaction directly into the CLIP visual encoder.Affinity-Weighted Gradient Propagation distributes contrastive learning signals to multiple high-affinity frames under weak supervision.
- Experiments on four PRVR benchmarks show state-of-the-art performance with fewer trainable parameters and lower video-encoder computation.
- Additional analyses show more temporally grounded frame representations and better generalization across datasets.
Limitations
The method may inherit CLIP’s representation biases, which can be reinforced by video-level supervision without precise frame-level labels. Such biases may favor visually aligned frames that are not the most temporally accurate evidence.
- ITA may inherit CLIP representation biases because training uses video-level supervision without precise frame-level labels.
- These biases can cause the model to favor frames highly aligned with CLIP but not necessarily the most temporally accurate evidence.Overcoming such biases under weak supervision remains an important direction for future work.
Potential Risks
Improved video retrieval can make large collections easier to search, but it may also enable surveillance, privacy-invasive search, and large-scale indexing of sensitive content. Responsible deployment should address privacy, consent, and safeguards against unintended use.
- Improved retrieval models can help users search large video collections more efficiently but may also be misused for surveillance and privacy-invasive search.
- Responsible deployment should consider data privacy, consent, and safeguards against unintended use.