PaperlayerView exact revision on arXiv

Source-linked AI summary

Text-Video Retrieval via Multi-Dimensional Saliency Assessment and Granularity-Aware Query Decomposition

Shuquan Wei, Xi Chen, Xu Chen, Xiangyang Jia

arXiv:2609.38949v1cs.IR

TL;DR

Text-video retrieval must handle redundant spatiotemporal video content, while prior single-aspect importance criteria and global text embeddings leave visual saliency and multi-level semantics insufficiently represented. MMTI combines saliency-aware key-feature selection with sentence-, frame-, and patch-level query interaction, and outperforms state-of-the-art methods on four standard benchmarks.

  • Problem

    Text-video retrieval methods often assess visual importance by a single aspect and encode text globally, despite videos’ multifaceted spatiotemporal content and captions’ multi-level semantics.

    Source: Introduction · pp. 1–2

  • Method

    MMTI uses KFS to select informative frames and patches with multidimensional saliency and learnable importance scores, and TVIM dynamically decomposes text into sentence, frame, and patch queries.

    Source: Introduction · p. 2

  • Results

    MMTI outperforms state-of-the-art methods on four standard benchmarks.

    Source: Introduction · p. 2, Conclusions · p. 7

  • Takeaways & Limitations

    Hierarchical alignment captures correspondences across levels and can reduce the impact of textual inaccuracies at any single granularity.

    Source: Introduction · p. 2

Abstract

from arXiv · show

Text-video retrieval, which aims to bridge visual and textual modalities by learning a joint embedding space, has become a crucial task in multimodal intelligence. Despite extensive efforts to mitigate visual redundancy, previous methods typically rely on a single-aspect criterion to assess visual importance, overlooking the multifaceted spatiotemporal nature of video. In addition, encoding text into a single global embedding to align with videos compresses temporal events and spatial entities into a unified representation space, further aggravating cross-modal misalignment. To address these issues, we propose MMTI, a method that jointly mitigates visual redundancy and enables multi-grained text-video interaction to achieve accurate multi-grained semantic alignment. Specifically, a key feature selection (KFS) mechanism adaptively identifies and aggregates informative frames and patches by jointly evaluating multi-dimensional saliency and learnable importance scores, effectively compacting dense visual features and mitigating visual redundancy. Furthermore, our proposed multi-grained text-video interaction module (TVIM) employs a dynamic gating mechanism to decompose the text query into sentence, frame, and patch queries (SFP), enabling multi-grained text-video alignment. Complementary alignment at different granularities is thereby achieved. Extensive experiments on four standard benchmarks demonstrate that our method outperforms state-of-the-art methods.

Introduction

Text-video retrieval must handle videos’ complex spatiotemporal content, yet prior approaches assess visual importance through a single aspect or compress text into a global embedding. MMTI combines multidimensional key-feature selection with multi-grained text-video alignment, and the paper reports improved retrieval accuracy over state-of-the-art methods on four benchmarks.

Introduction

  • Video’s dynamic objects, actions, and events make accurate cross-modal alignment and efficient retrieval difficult.Videos have complex spatiotemporal structures and high temporal redundancy.Source: Introduction · p. 1
  • Prior methods often assess visual importance using a single criterion or compress text into one global embedding, limiting multi-level alignment.The global representation unifies temporal events and spatial entities, while captions contain multi-level semantics.Source: Introduction · p. 2
  • KFS jointly evaluates multidimensional saliency and learnable importance scores, then uses soft assignment to compact visual features while retaining discriminative semantics.It adaptively selects informative frames and patches.Source: Introduction · p. 2
  • MMTI outperforms state-of-the-art methods on four standard benchmarks.The method jointly addresses visual redundancy and multi-grained text-video alignment.Source: Introduction · p. 2
  • TVIM dynamically decomposes text into sentence, frame, and patch queries for alignment at complementary semantic levels.The proposed hybrid objective combines bidirectional contrastive learning with multi-grained consistency regularization.Source: Introduction · p. 2

Related Work

Prior text-video retrieval work extends pretrained image-text models through frame aggregation and text interaction, while redundancy and multi-grained alignment remain concerns. The related work also describes methods for enriching video representations, but notes that they do not exploit captions’ inherent multi-level semantics.

Text-Video Retrieval

  • CLIP-based video methods use frame features and aggregation strategies, but processing all visual features can introduce redundancy [Gorti et al. 2022].Other approaches address redundancy through pruning or frame-adaptation mechanisms, often using a single-aspect importance criterion.Source: Text-Video Retrieval · p. 2

Enhance Video Features from Text Interaction

  • Text-interaction methods include multi-grained attention and global-local mechanisms, while visual debiasing and narration enrichment target additional representation gaps [Ma et al. 2022; Yang et al. 2024; Le et al. 2025; Hur et al. 2025].The cited discussion says these approaches do not exploit captions’ inherent multi-level semantics [Jin et al. 2023c].Source: Enhance Video Features from Text… · pp. 2–3
  • MMTI’s overview pairs KFS selection of informative frames and patches with TVIM’s sentence-, frame-, and patch-query alignment.KFS uses multidimensional saliency and learnable importance scores; TVIM aligns the decomposed queries with selected visual features.Source: Figure 2 · p. 3

Method

MMTI retrieves videos by combining compacted key visual features with text queries aligned at sentence, frame, and patch granularities. Its training objective combines bidirectional contrastive learning with consistency regularization.

Problem Definition

  • The retrieval objective is to find the video most semantically relevant to a text query in a directly comparable joint embedding space.Videos are represented as frame sequences, with each frame divided into patches, and text queries as word sequences.Source: Problem Definition · p. 3

Overview

  • KFS compacts video features while preserving key semantics; TVIM aligns decomposed text queries with key frame and patch features, and training combines contrastive loss with consistency regularization.At inference, fused similarity scores are used for retrieval.Source: Overview · p. 3

Key Feature Selection

  • Frame selection combines multi-dimensional saliency with learned importance, selects top-scoring anchors, and fuses original frame features through soft assignment.Frame saliency reflects novelty, semantic distinctiveness, and energy variance; anchor importance informs the assignment.Source: Key Feature Selection · p. 4
  • Patch selection combines multi-dimensional saliency with learned scores, selects the highest-scoring anchor patches, and generates key patch features through soft assignment fusion.Patch saliency includes distance from the center patch, feature magnitude, and similarity to the left neighbor.Source: Key Feature Selection · p. 4

Text-Video Interaction Module

  • TVIM uses a dynamic gate to fuse the sentence embedding with a global word summary, then derives sentence, frame, and patch queries for different-granularity alignment.The frame and patch queries are constructed from the refined sentence embedding and word summary in different orders.Source: Text-Video Interaction Module · pp. 4–5
  • Frame queries use confidence-modulated attention over key frames, while patch queries follow an analogous interaction over key patches using chunk-wise computation.The frame-level context is obtained by weighted summation; chunking handles the larger number of patches.Source: Text-Video Interaction Module · p. 5

Objective Function

  • The objective combines bidirectional InfoNCE with consistency regularization that encourages fused similarity to match the ensemble of frame- and patch-level similarities.The consistency loss uses KL divergence and a temperature parameter.Source: Objective Function · p. 5

Experiments

Experiments across four benchmarks report state-of-the-art retrieval performance for MMTI under the ViT-B/32 backbone. Ablations and selection-ratio analyses examine its components and visual compression settings, while qualitative examples show selected frames and patches corresponding to query semantics.

Experimental Settings

  • Experiments use MSR-VTT, MSVD, ActivityNet Captions, and DiDeMo, with standard R@1, R@5, R@10, and Rsum metrics.The study uses CLIP ViT-B/32 features and, unless specified otherwise, selection ratios of 0.67 for frames and 0.55 for patches.Source: Experimental Settings · p. 5

Comparison with State-of-the-Art Methods

Ablation Study

  • Replacing SFP with a single global text embedding lowers R@1 from 56.6% to 48.0% and Rsum from 241.0 to 217.6.The authors state that three query granularities align with the hierarchical structure of video frames and patches.Source: Ablation Study · p. 6, Table 3 · p. 6
  • Without KFS, R@1 is 50.2% and Rsum is 221.4; removing novelty or semantic distinctiveness yields 47.7% R@1, removing energy variance 47.2%, and Hard TopK 48.4%.The full model records 56.6% R@1; the authors describe the saliency components as complementary and the soft assignment fusion as important for aggregation.Source: Ablation Study · pp. 6–7, Table 3 · p. 6
  • Removing the consistency loss lowers R@1 to 54.1% and Rsum to 235.5; the authors report the full model performs best across all metrics.They associate KL divergence regularization with maintaining consistent cross-modal alignment across granularities.Source: Table 3 · p. 6, Ablation Study · p. 7

KFS Selection Ratio Analysis

  • The frame ratio 0.67 and patch ratio 0.55 achieve 56.6% R@1; overly aggressive selection at 0.20 and 0.10 drops R@1 to 45.8%.Moderate ratio ranges outperform the no-KFS baseline, and the selected optimal configuration is used across all four benchmarks without per-dataset tuning.Source: KFS Selection Ratio Analysis · p. 7, Figure 4 · p. 7

Qualitative Analysis

  • Across three query examples, KFS selects frames and patches corresponding to named people, objects, interactions, and attributes while suppressing irrelevant background patches.The examples include a classroom interaction, a multi-entity sheep scene, and a child, bird, and box.Source: Qualitative Analysis · p. 7, Figure 5 · p. 7

Conclusions

MMTI jointly mitigates visual redundancy and enables multi-grained text-video interaction. Its KFS selects informative frames and patches, while TVIM decomposes queries for alignment with selected visual features at complementary semantic levels; experiments on four benchmarks report performance above the state of the art.

Conclusions

  • MMTI combines learnable key feature selection of informative frames and patches with query decomposition into sentence, frame, and patch levels, and outperforms state-of-the-art methods on four benchmarks.KFS jointly evaluates multi-dimensional saliency and learnable importance scores; TVIM aligns the decomposed query with selected visual features.Source: Conclusions · p. 7
Loading 2609.38949v1…