Source-linked AI summary
Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning
Shizhe Chen, Yida Zhao, Qin Jin, Qi Wu
TL;DR
Fine-grained video-text retrieval remains challenging because keyword systems and simple joint embeddings do not adequately represent detailed semantics and component relationships. HGR decomposes videos and texts into event, action, and entity levels, applies attention-based graph reasoning, and aggregates hierarchical matches. The model reports superior results across three datasets, better generalization to an unseen dataset, and improved discrimination of subtle semantic differences.
Problem
Keyword retrieval and simple joint embeddings inadequately capture fine-grained semantics and relationships in complicated video-text content.
Method
HGR decomposes video-text matching into event, action, and entity levels, reasons over textual semantic-role graphs, aligns corresponding video representations, and aggregates level-wise matches.
Results
HGR reports superior experimental results on three video-text datasets, better generalization on an unseen dataset, and improved recognition of fine-grained semantic changes.
Takeaways & Limitations
Hierarchical decomposition supports video-text matching that covers both global and local details and distinguishes comprehensive descriptions from correct but incomplete ones.
Takeaways & Limitations
Using separate video embeddings rather than directly parsing videos into hierarchical structures avoids the challenges of temporal segmentation, object detection, and tracking.
Abstract
from arXiv · showhide
Cross-modal retrieval between videos and texts has attracted growing attentions due to the rapid emergence of videos on the web. The current dominant approach for this problem is to learn a joint embedding space to measure cross-modal similarities. However, simple joint embeddings are insufficient to represent complicated visual and textual details, such as scenes, objects, actions and their compositions. To improve fine-grained video-text retrieval, we propose a Hierarchical Graph Reasoning (HGR) model, which decomposes video-text matching into global-to-local levels. To be specific, the model disentangles texts into hierarchical semantic graph including three levels of events, actions, entities and relationships across levels. Attention-based graph reasoning is utilized to generate hierarchical textual embeddings, which can guide the learning of diverse and hierarchical video representations. The HGR model aggregates matchings from different video-text levels to capture both global and local details. Experimental results on three video-text datasets demonstrate the advantages of our model. Such hierarchical decomposition also enables better generalization across datasets and improves the ability to distinguish fine-grained semantic differences.
1. Introduction
Fine-grained video-text retrieval is difficult because keyword systems and compact or sequential joint representations struggle with detailed semantics and relationships. HGR addresses this by decomposing matching into event, action, and entity levels, reasoning over their interactions, and aggregating hierarchical matches.
- Keyword-based retrieval struggles to distinguish fine-grained contents such as a white dog chasing a black cat.
- Joint embedding methods capture salient meanings but compact representations can miss detailed actions, entities, and their semantic roles.
- Sequential frame-and-word representations can neglect topological relations among local components, while video-text pairs are more weakly supervised than image-text pairs.
- HGR decomposes matching into global events, local actions, and entities, using attention-based graph reasoning to model their cross-level interactions.
- HGR aligns hierarchical video and text components and aggregates matching scores across all three levels for broader fine-grained semantic coverage.
- Experiments on MSR-VTT, TGIF, and VATEX report consistent improvements, better unseen-dataset generalization, and stronger recognition of subtle semantic changes.
2. Related Works
Prior work mainly uses common-space embeddings for image-text matching and increasingly explores graph-based reasoning and phrase-level disentanglement. HGR extends these directions to hierarchical semantic graphs for full-sentence video-text matching.
- Most image-text matching methods encode images and sentences as fixed-dimensional vectors in a common latent space for similarity measurement.
- Prior fine-grained action retrieval disentangles action phrases by parts of speech, but HGR targets the greater complexity of full sentences.
- HGR combines hierarchical sentence decomposition with graph-based reasoning to integrate video-text matching at multiple semantic levels.
3. Hierarchical Graph Reasoning Model
HGR represents video-text pairs at hierarchical global-to-local levels, using semantic-role graph reasoning for texts and corresponding multi-level video embeddings. It aggregates global and local matches into an overall similarity learned with weakly supervised video-text pairs.
- Hierarchical textual encoding: HGR constructs semantic role graphs with a sentence-level event node connected to action and entity nodes.Actions are obtained from verbs and entities from noun phrases; graph connections represent how local nodes compose the global event.
- Hierarchical textual encoding: Bi-LSTM word representations and attention produce the global event embedding, while max pooling produces action and entity node representations.The resulting textual levels are represented as global event, action-node, and entity-node embeddings.
- Hierarchical textual encoding: Attention-based graph reasoning selects relevant neighbor contexts and uses semantic-role-specific transformations to enhance node representations.Factorizing transformations into shared and role-specific components reduces parameters while retaining role awareness; final outputs are ce, ca, and co.
- Hierarchical video encoding: Because directly parsing videos into hierarchical structures is challenging, HGR builds three independent video embeddings focused on events, actions, and entities.The event level uses one attention-pooled global vector, while action and entity levels use frame-wise feature sequences.
- Video-text matching: Global and local similarities are aggregated across the three levels, with local text nodes dynamically aligned to video frames through attention.The final video-text similarity averages cross-modal similarities at all levels and is trained with contrastive ranking loss using hard negatives.
4. Experiments
Experiments evaluate HGR across in-domain, cross-dataset, ablation, and fine-grained retrieval settings. Results show consistent gains from hierarchical representations, graph reasoning, and global-to-local matching, while qualitative cases illustrate both strengths and failure modes.
- Experimental settings: HGR is evaluated against state-of-the-art methods on MSR-VTT, TGIF, and VATEX using text-to-video and video-to-text retrieval.The evaluation uses standard retrieval metrics including R@K, MedR, MnR, and rsum.
- Comparison with State of The Arts: 19.5% and 15.4% relative gains on R@1 are reported over Dual Encoding for text-to-video and video-to-text retrieval, respectively, on MSR-VTT.The overall rsum metric increases by +23.8, and HGR uses half as many parameters and computations as Dual Encoding.
- Comparison with State of The Arts: Consistent improvements over state-of-the-art models are obtained on TGIF and VATEX with different video features.The results support decomposing videos and texts into global-to-local hierarchical graph structures.
- Ablation Studies: 0.9 and 1.7 R@10 reductions occur when graph attention is replaced by neighbor averaging for text-to-video and video-to-text retrieval, respectively.Role awareness also benefits graph reasoning by modeling how components relate within an event; attention shifts from main arguments to temporal arguments and global-event context across layers.
- Ablation Studies: 172.4 versus 167.9 rsum shows the benefit of hierarchical video embeddings over the ablated representation.The global event level performs best alone, while combinations of global and local levels improve retrieval because the levels are complementary.
- Fine-grained Binary Selection: HGR achieves its largest binary-selection improvement on incomplete events and best distinguishes entity replacements, especially scene replacements.It outperforms VSE++ by 4.87% on role switching but is slightly inferior to Dual Encoding, partly because Youtube2Text descriptions average seven words.
5. Conclusion
HGR addresses fine-grained video-text retrieval by decomposing videos and texts into hierarchical event, action, and entity levels. It uses attention-based graph reasoning and multi-level matching to achieve strong retrieval, generalization, and semantic discrimination.
- HGR decomposes videos and texts into hierarchical semantic levels of events, actions, and entities.
- Attention-based graph reasoning generates hierarchical textual embeddings that align with videos at different levels.
- The model aggregates matching scores from different levels to capture both global and local details.
- Experiments demonstrate superior results across three video-text datasets, better generalization to unseen data, and fine-grained semantic discrimination.
A. Semantic Roles
The semantic role graph parses text into a global event node, action nodes, and entity nodes. Typed edges connect events to actions and actions to entities according to semantic roles.
- Text is parsed into a hierarchical semantic role graph with global event, action, and entity nodes.
- Action nodes connect to the global event node through action edge types.
- Entity nodes connect to corresponding action nodes through edge types determined by their semantic roles.
- The graph uses semantic roles and descriptions defined with reference to linguistic experts.
B. Binary Selection Task
The binary selection task evaluates fine-grained text discrimination by requiring a model to choose the better-matching sentence from two similar but semantically different alternatives.
- The task asks models to select the sentence that better matches a given video.
- Each test video uses one randomly selected ground-truth description as the positive sentence.
- The negative sentence is generated by perturbing the ground-truth description to create a semantically different alternative.
- Figure 6 reports cross-modal video-text retrieval results on the TGIF and VATEX testing sets.
C. Additional Qualitative Examples
Qualitative retrieval examples on TGIF and VATEX show robust and superior performance by HGR across different datasets.
- HGR achieves robust and superior cross-modal retrieval performance on TGIF and VATEX examples.