Source-linked AI summary
HVD: Human Vision-Driven Video Representation Learning for Text-Video Retrieval
Zequn Xie, Xin Liu, Boyun Zhang, Yuxiao Lin, Sihang Cai, Tao Jin
TL;DR
Text-video retrieval methods can blindly interact with irrelevant visual features when queries are sparse. HVD uses human-inspired frame selection and patch compression for coarse-to-fine alignment, achieving state-of-the-art retrieval performance across five benchmarks.
Problem
Sparse textual queries make it difficult for existing text-video retrieval methods to distinguish relevant visual information from background or irrelevant frames during feature interaction.
Method
HVD combines FFSM for similarity-based key-frame selection with PFCM for clustering, attention-based patch compression, and fine-grained entity alignment.
Results
HVD achieves state-of-the-art retrieval performance on five benchmark datasets, including MSRVTT, DiDeMo, LSMDC, ActivityNet, and Charades.
Takeaways & Limitations
The results support jointly improving video representations at frame and patch levels through human-inspired coarse- and fine-grained alignment.
Abstract
from arXiv · showhide
The success of CLIP has driven substantial progress in text-video retrieval. However, current methods often suffer from "blind" feature interaction, where the model struggles to discern key visual information from background noise due to the sparsity of textual queries. To bridge this gap, we draw inspiration from human cognitive behavior and propose the Human Vision-Driven (HVD) model. Our framework establishes a coarse-to-fine alignment mechanism comprising two key components: the Frame Features Selection Module (FFSM) and the Patch Features Compression Module (PFCM). FFSM mimics the human macro-perception ability by selecting key frames to eliminate temporal redundancy. Subsequently, PFCM simulates micro-perception by aggregating patch features into salient visual entities through an advanced attention mechanism, enabling precise entity-level matching. Extensive experiments on five benchmarks demonstrate that HVD not only captures human-like visual focus but also achieves state-of-the-art performance.
1 Zhejiang University 2 Southwestern University of Finance and Economics
The paper is positioned within multimodal representation learning, text-video retrieval, and feature enhancement and interaction.
- HVD addresses text-video retrieval through multimodal representation learning and feature enhancement and interaction.
1. INTRODUCTION
Existing text-video retrieval methods use shared CLIP representations and feature enhancement, but sparse queries can cause blind interaction with irrelevant visual content. HVD addresses this through coarse-to-fine, human-inspired frame selection and patch compression.
- CLIP-based methods project text and video into a shared latent space for feature-level similarity relationships.
- Sparse text queries can leave existing methods interacting with irrelevant frames, such as interview scenes unrelated to baseball.
- Human-inspired alignment first selects text-relevant key frames, then extracts visual entities for repeated keyword comparison.
- Prior coarse-to-fine and feature-enhancement methods improve retrieval but are described as failing to capture human interactive cognition and suffering blind feature interaction.
- FFSM selects similarity-based key frames, removes redundant or irrelevant frames, and retains patches needed for fine-grained alignment.
2. METHODS
HVD uses CLIP features and a coarse-to-fine pipeline that selects relevant frames, compresses their patches into salient entities, and jointly trains global and local similarities for retrieval.
- Preliminaries: CLIP extracts patch and frame features from videos and word and sentence features from text queries.
- Preliminaries: Feature interaction computes similarity between textual and visual features, with contrastive loss maximizing positive-pair similarity and minimizing negative-pair similarity.
- Frame Features Selection Module: FFSM compares sentence features with frame features and selects the top-N* relevant frames by text-frame similarity.
- Patch Features Compression Module: PFCM compresses patches from selected frames using DPC-KNN clustering, then merges neighboring patches around high-density, high-distance cluster centers.
- Patch Features Compression Module: PFCM uses attention to re-represent merged patches, focusing on key features and spatial relationships while reducing patch count.
- Training and Inference: The framework combines FFSM frame features with PFCM patch features, jointly minimizes global and local losses, and aggregates both similarities for retrieval ranking.
3. EXPERIMENTS
HVD is evaluated across five benchmarks using retrieval metrics, ablations, and visualizations. Results show strong retrieval performance, benefits from combining frame selection with patch compression, and best performance at 0.50 retention for both frames and patches.
- Experimental Setup: HVD is evaluated on MSRVTT, DiDeMo, LSMDC, ActivityNet, and Charades using R@K, MdR, and MnR.Experiments use CLIP as the backbone, 12 sampled frames, and 0.5 frame and patch retention ratios.
- Retrieval Performance: 48.8 R@1 on MSRVTT improves by 0.2 over HBI, indicating gains from visual feature selection and compression.The reported comparison is for MSRVTT retrieval performance.
- Retrieval Performance: HVD achieves consistent performance improvements on both the long-video DiDeMo and short-text LSMDC datasets.
- Ablation Studies: Combining macro frame selection with micro patch compression achieves the best retrieval performance in the FFSM and PFCM ablation study.The ablation compares the baseline with frame selection and patch compression configurations.
- Ablation Studies: FFSM provides refined information to PFCM, while PFCM complements FFSM with additional details.
- Ablation Studies: Setting frame and patch retention ratios to 0.50 yields the best retrieval performance, while later PFCM operations progressively reduce interacting features.The frame ratio retains video frames and the patch ratio retains image patches.
4. CONCLUSION
The paper concludes that HVD combines frame selection and patch compression to improve text–video retrieval through coarse- and fine-grained alignment. Across five benchmarks, it reports state-of-the-art retrieval performance and validates the modules through ablations and visualizations.
- HVD integrates FFSM and PFCM to improve video representations at frame and patch levels for coarse- and fine-grained text–video alignment.
- HVD achieves state-of-the-art retrieval performance across MSRVTT, DiDeMo, LSMDC, ActivityNet, and Charades.
- Ablation studies, hyper-parameter evaluations, and visualizations validate the effectiveness and intuitiveness of the proposed modules.