PaperlayerView exact revision on arXiv

Source-linked AI summary

Delving Deeper: Hierarchical Visual Perception for Robust Video-Text Retrieval

Zequn Xie, Boyun Zhang, Yuxiao Lin, Tao Jin

arXiv:2601.12768v1cs.CVcs.MM

TL;DR

Video-text retrieval is limited by redundancy and reliance on coarse final-layer features, which overlook intermediate hierarchical semantics. HVP-Net extracts and refines multi-layer visual features, then aligns them with text across granularities; it achieves state-of-the-art results across MSR-VTT, DiDeMo, and ActivityNet.

  • Problem

    VTR methods predominantly rely on final-layer vision features and overlook hierarchical semantics in intermediate layers.

    Source: §1 · p. 1

  • Method

    HVP-Net extracts features from multiple intermediate vision-encoder layers, refines patch features, and optimizes multi-granularity alignment with contrastive learning.

    Source: §2 · p. 1, §2.3 · p. 3

  • Results

    HVP-Net establishes new state-of-the-art results on MSR-VTT, DiDeMo, and ActivityNet, including 56.7% R@1 on MSR-VTT.

    Source: §3.2 · pp. 3–4, §4 · p. 4

  • Takeaways & Limitations

    Mining internal hierarchical features of pre-trained models is a potent strategy for cross-modal alignment in video-text retrieval.

    Source: §4 · p. 4

Abstract

from arXiv · show

Video-text retrieval (VTR) aims to locate relevant videos using natural language queries. Current methods, often based on pre-trained models like CLIP, are hindered by video's inherent redundancy and their reliance on coarse, final-layer features, limiting matching accuracy. To address this, we introduce the HVP-Net (Hierarchical Visual Perception Network), a framework that mines richer video semantics by extracting and refining features from multiple intermediate layers of a vision encoder. Our approach progressively distills salient visual concepts from raw patch-tokens at different semantic levels, mitigating redundancy while preserving crucial details for alignment. This results in a more robust video representation, leading to new state-of-the-art performance on challenging benchmarks including MSRVTT, DiDeMo, and ActivityNet. Our work validates the effectiveness of exploiting hierarchical features for advancing video-text retrieval. Our codes are available at https://github.com/boyun-zhang/HVP-Net.

1. INTRODUCTION

VTR methods commonly rely on final-layer vision features, overlooking hierarchical semantics in intermediate layers. HVP-Net addresses this by processing shallow, mid, and deep layers to capture concepts from textures to events.

  • Current VTR approaches predominantly rely on final-layer vision features, overlooking rich hierarchical semantics in intermediate layers.Source: §1 · p. 1
  • HVP-Net processes visual data through shallow, mid, and deep layers to extract concepts ranging from low-level textures to high-level events.Source: Figure 1 · p. 1

2. METHODOLOGY

HVP-Net builds video representations from hierarchical frame and patch features, refines redundant patch tokens, and aligns text with video at multiple granularities. Layer-wise similarity scores and contrastive optimization support retrieval across semantic levels.

Hierarchical Feature Encoding

  • HVP-Net extracts frame and patch features from multiple intermediate vision-encoder layers instead of relying only on final-layer outputs.Frame features come from [CLS] tokens, while patch features come from patch tokens for each selected layer.Source: §2 · p. 1, §2.1 · p. 2

Multi-layer Patch Processing

  • The Multi-layer Patch Processing module repeatedly compresses redundant patch tokens into salient concepts and refines them through cross-attention to original tokens.Density-Peak Clustering identifies K cluster centers from M tokens, followed by saliency-weighted merging and attention-based enrichment.Source: §2.2 · p. 2, §2.3 · p. 3
  • The refined patch features form a purified multi-level representation of visual entities for downstream alignment.Source: §2.2 · p. 2

Multi-granularity Alignment

  • Text aligns with video through sentence-frame, sentence-patch, and word-patch similarities across all extracted feature levels.Patch-level and word-level matching uses max-pooling followed by learnable weighted aggregation, with symmetric patch-to-word scoring averaged into the final score.Source: §2.3 · p. 2
  • The total objective sums bidirectional contrastive losses across alignment granularities and feature levels, while inference sums similarity scores across layers and granularities.Source: §2.3 · p. 3

3. EXPERIMENTS

HVP-Net is evaluated on three video-text retrieval benchmarks against recent methods, with experiments showing strong cross-task performance and that its multi-layer processing, layer selection, and alignment losses are important.

Experimental Setup

  • Experiments use MSR-VTT, DiDeMo, and ActivityNet Captions with Recall@K, Median Rank, and Mean Rank for text-to-video and video-to-text retrieval.MSR-VTT uses the standard 1k-A test split.Source: §3.1 · p. 3

Comparison with State-of-the-Art Methods

  • HVP-Net is compared with recent state-of-the-art methods, including CLIP4Clip, X-CLIP, BiHSSP, and MUSE.Source: §3.2 · p. 3
  • 56.7% R@1 on MSR-VTT text-to-video retrieval surpasses MUSE by 5.8%, with leading performance across metrics and both retrieval directions.Source: §3.2 · p. 3
  • HVP-Net establishes new state-of-the-art results on DiDeMo and ActivityNet, outperforming prior works across all reported recall metrics.Source: §3.2 · p. 4, Table 2 · p. 4

Ablation Studies and Analysis

  • The full model improves T2V R@1 from 54.2% with final-layer features to 56.7%, whereas raw intermediate-feature aggregation without MPP collapses to 22.1%.The ablation evaluates core components on MSR-VTT.Source: §3.3 · p. 4
  • Sampling layers 1, 6, and 12 reaches 56.7% T2V R@1, outperforming the last three layers at 55.6% and five-layer sampling at 51.9%.Source: §3.3 · p. 4
  • Removing sentence-frame, sentence-patch, or word-patch alignment causes drops of 8.5, 3.5, and 23.4 points, respectively.The results identify word-patch alignment as the most impactful of the three supervision levels.Source: §3.3 · p. 4

4. CONCLUSION

The paper addresses redundancy in final-layer-only text-video retrieval by introducing HVP-Net and its multi-layer patch processing framework. It reports state-of-the-art benchmark performance and identifies adaptive layer selection and video-to-text improvements as future directions.

  • HVP-Net addresses information redundancy in conventional methods that rely solely on final-layer features.Source: §4 · p. 4
  • The Multi-layer Patch Processing module refines intermediate-layer vision features to distill salient concepts and create a more robust cross-modal embedding space.Source: §4 · p. 4
  • 56.7% R@1 on MSR-VTT is reported alongside new state-of-the-art performance on multiple benchmarks.Source: §4 · p. 4
  • Future work could explore adaptive layer-selection strategies and mechanisms specifically improving video-to-text retrieval.Source: §4 · p. 4

Related papers

Loading 2601.12768v1…