Source-linked AI summary
Delving Deeper: Hierarchical Visual Perception for Robust Video-Text Retrieval
Zequn Xie, Boyun Zhang, Yuxiao Lin, Tao Jin
TL;DR
Video-text retrieval is limited by redundancy and reliance on coarse final-layer features, which overlook intermediate hierarchical semantics. HVP-Net extracts and refines multi-layer visual features, then aligns them with text across granularities; it achieves state-of-the-art results across MSR-VTT, DiDeMo, and ActivityNet.
Problem
VTR methods predominantly rely on final-layer vision features and overlook hierarchical semantics in intermediate layers.
Method
HVP-Net extracts features from multiple intermediate vision-encoder layers, refines patch features, and optimizes multi-granularity alignment with contrastive learning.
Results
HVP-Net establishes new state-of-the-art results on MSR-VTT, DiDeMo, and ActivityNet, including 56.7% R@1 on MSR-VTT.
Takeaways & Limitations
Mining internal hierarchical features of pre-trained models is a potent strategy for cross-modal alignment in video-text retrieval.
Abstract
from arXiv · showhide
Video-text retrieval (VTR) aims to locate relevant videos using natural language queries. Current methods, often based on pre-trained models like CLIP, are hindered by video's inherent redundancy and their reliance on coarse, final-layer features, limiting matching accuracy. To address this, we introduce the HVP-Net (Hierarchical Visual Perception Network), a framework that mines richer video semantics by extracting and refining features from multiple intermediate layers of a vision encoder. Our approach progressively distills salient visual concepts from raw patch-tokens at different semantic levels, mitigating redundancy while preserving crucial details for alignment. This results in a more robust video representation, leading to new state-of-the-art performance on challenging benchmarks including MSRVTT, DiDeMo, and ActivityNet. Our work validates the effectiveness of exploiting hierarchical features for advancing video-text retrieval. Our codes are available at https://github.com/boyun-zhang/HVP-Net.
1. INTRODUCTION
VTR methods commonly rely on final-layer vision features, overlooking hierarchical semantics in intermediate layers. HVP-Net addresses this by processing shallow, mid, and deep layers to capture concepts from textures to events.
- Current VTR approaches predominantly rely on final-layer vision features, overlooking rich hierarchical semantics in intermediate layers.
- HVP-Net processes visual data through shallow, mid, and deep layers to extract concepts ranging from low-level textures to high-level events.
2. METHODOLOGY
HVP-Net builds video representations from hierarchical frame and patch features, refines redundant patch tokens, and aligns text with video at multiple granularities. Layer-wise similarity scores and contrastive optimization support retrieval across semantic levels.
- Hierarchical Feature Encoding: HVP-Net extracts frame and patch features from multiple intermediate vision-encoder layers instead of relying only on final-layer outputs.Frame features come from [CLS] tokens, while patch features come from patch tokens for each selected layer.
- Multi-layer Patch Processing: The Multi-layer Patch Processing module repeatedly compresses redundant patch tokens into salient concepts and refines them through cross-attention to original tokens.Density-Peak Clustering identifies K cluster centers from M tokens, followed by saliency-weighted merging and attention-based enrichment.
- Multi-layer Patch Processing: The refined patch features form a purified multi-level representation of visual entities for downstream alignment.
- Multi-granularity Alignment: Text aligns with video through sentence-frame, sentence-patch, and word-patch similarities across all extracted feature levels.Patch-level and word-level matching uses max-pooling followed by learnable weighted aggregation, with symmetric patch-to-word scoring averaged into the final score.
- Multi-granularity Alignment: The total objective sums bidirectional contrastive losses across alignment granularities and feature levels, while inference sums similarity scores across layers and granularities.
3. EXPERIMENTS
HVP-Net is evaluated on three video-text retrieval benchmarks against recent methods, with experiments showing strong cross-task performance and that its multi-layer processing, layer selection, and alignment losses are important.
- Experimental Setup: Experiments use MSR-VTT, DiDeMo, and ActivityNet Captions with Recall@K, Median Rank, and Mean Rank for text-to-video and video-to-text retrieval.MSR-VTT uses the standard 1k-A test split.
- Comparison with State-of-the-Art Methods: HVP-Net is compared with recent state-of-the-art methods, including CLIP4Clip, X-CLIP, BiHSSP, and MUSE.
- Comparison with State-of-the-Art Methods: 56.7% R@1 on MSR-VTT text-to-video retrieval surpasses MUSE by 5.8%, with leading performance across metrics and both retrieval directions.
- Comparison with State-of-the-Art Methods: HVP-Net establishes new state-of-the-art results on DiDeMo and ActivityNet, outperforming prior works across all reported recall metrics.
- Ablation Studies and Analysis: The full model improves T2V R@1 from 54.2% with final-layer features to 56.7%, whereas raw intermediate-feature aggregation without MPP collapses to 22.1%.The ablation evaluates core components on MSR-VTT.
- Ablation Studies and Analysis: Sampling layers 1, 6, and 12 reaches 56.7% T2V R@1, outperforming the last three layers at 55.6% and five-layer sampling at 51.9%.
- Ablation Studies and Analysis: Removing sentence-frame, sentence-patch, or word-patch alignment causes drops of 8.5, 3.5, and 23.4 points, respectively.The results identify word-patch alignment as the most impactful of the three supervision levels.
4. CONCLUSION
The paper addresses redundancy in final-layer-only text-video retrieval by introducing HVP-Net and its multi-layer patch processing framework. It reports state-of-the-art benchmark performance and identifies adaptive layer selection and video-to-text improvements as future directions.
- HVP-Net addresses information redundancy in conventional methods that rely solely on final-layer features.
- The Multi-layer Patch Processing module refines intermediate-layer vision features to distill salient concepts and create a more robust cross-modal embedding space.
- 56.7% R@1 on MSR-VTT is reported alongside new state-of-the-art performance on multiple benchmarks.
- Future work could explore adaptive layer-selection strategies and mechanisms specifically improving video-to-text retrieval.