Source-linked AI summary
Cross-Modal and Hierarchical Modeling of Video and Text
Bowen Zhang, Hexiang Hu, Fei Sha
TL;DR
The paper studies how to preserve cross-modal correspondences in hierarchical video and text sequences, where videos comprise clips and paragraphs comprise sentences. It introduces hierarchical sequence embedding with discriminative alignment and layer-wise reconstruction, and reports improved retrieval performance plus utility for zero-shot action recognition and video description.
Problem
Less is known about learning shared semantic representations for complex videos and paragraphs while preserving their intrinsic cross-modal correspondences.
Method
Hierarchical sequence embedding jointly models videos and texts at multiple granularities, aligning paired clips and sentences and using layer-wise reconstruction objectives.
Results
The hierarchical models improve retrieval over flat baselines across metrics, with Recall@1 improving by over ∼15% using C3D and ∼30% using InceptionV3 over previous state-of-the-art results.
Takeaways & Limitations
The learned embeddings are useful across video-text retrieval, zero-shot action recognition, and video description tasks.
Takeaways & Limitations
The method lacks enough information to define the low-level matching loss precisely and leaves soft assignment of matching for future work.
Abstract
from arXiv · showhide
Visual data and text data are composed of information at multiple granularities. A video can describe a complex scene that is composed of multiple clips or shots, where each depicts a semantically coherent event or action. Similarly, a paragraph may contain sentences with different topics, which collectively conveys a coherent message or story. In this paper, we investigate the modeling techniques for such hierarchical sequential data where there are correspondences across multiple modalities. Specifically, we introduce hierarchical sequence embedding (HSE), a generic model for embedding sequential data of different modalities into hierarchically semantic spaces, with either explicit or implicit correspondence information. We perform empirical studies on large-scale video and paragraph retrieval datasets and demonstrated superior performance by the proposed methods. Furthermore, we examine the effectiveness of our learned embeddings when applied to downstream tasks. We show its utility in zero-shot action recognition and video captioning.
1 Introduction
The paper addresses the limited understanding of cross-modal correspondence in complex videos and paragraphs. It proposes jointly modeling their hierarchical structures and low-level correspondences.
- Clip-sentence embeddings are aligned in a shared local semantic space, while video-paragraph embeddings are aligned in a global semantic space.
- Multimodal learning has established shared semantic spaces for images, visual categories, and smaller linguistic units, but complex videos and paragraphs remain less studied.
- Videos contain coherent clips and paragraphs contain ordered descriptive sentences, yet flat frame-word modeling may obscure their temporal and semantic correspondences.
- The proposed approach jointly models video and text with hierarchical sequence learning while preserving correspondences across granularities.
- The paper evaluates the approach on retrieval, zero-shot action recognition, and video description, followed by experimental analyses and ablation studies.
2 Related Work
Prior work studied hierarchical modeling in individual modalities and cross-modal embeddings for simpler data. This paper differs by modeling videos and texts hierarchically together while bridging multiple granularities.
- Hierarchical Sequence Embedding Models: Related work includes hierarchical auto-encoding for paragraphs and hierarchical video modeling paired with one-layer caption generation.
- Hierarchical Sequence Embedding Models: The proposed approach models both modalities hierarchically and learns parameters using cross-modal correspondences.
- Cross-modal Embedding Learning: Earlier cross-modal methods aligned images with visual categories, words, or noun phrases, whereas this work connects videos and texts at multiple granularities.
- Action Recognition in Videos: For action recognition, the learned video semantic embedding matches holistic video representations to text representations to assess generalization.
- Flat sequence modeling treats a video or paragraph as a sequence of frames or words and ignores their hierarchical structures.
3 Approach
The approach jointly models videos and paragraphs as hierarchical sequences, aligning their global and local semantics through discriminative matching and layer-wise reconstruction. It also extends low-level matching to settings with only coarse video–paragraph correspondence.
- 3.1 Settings and Notations: Videos are represented as clips containing frames, while paragraphs contain sentences composed of words, with paired clips and sentences in the strong-correspondence setting.The framework derives clip and sentence embeddings from lower-level features, then aggregates them into video and paragraph embeddings.
- 3.2 Flat Sequence Modeling: Flat sequence modeling can struggle with long-range dependencies because its LSTM/GRU layer may require many units, complicating optimization.The hierarchical formulation addresses this deficiency by separately modeling frames, clips, words, and sentences.
- 3.3 Hierarchical Sequence Modeling: Hierarchical sequence embedding models both modalities hierarchically, using lower-level embeddings as inputs to higher-level sequence encoders.GRU-based encoder–decoder structures produce embeddings for the nested video and text sequences.
- 3.3 Hierarchical Sequence Modeling: The model learns cross-modal alignment with margin-based losses at both low and high levels, separating matched video–text pairs from unmatched pairs.The high-level and low-level objectives correspond to video–paragraph and clip–sentence matching, respectively.
- 3.4 Final Learning Objective and Its Extensions: Layer-wise reconstruction adds decoders that reconstruct clips or sentences from high-level embeddings and frames or words from low-level embeddings.The reconstruction loss minimizes differences between decoder-generated sequences and encoder-computed data representations.
- 3.4 Final Learning Objective and Its Extensions: With only high-level correspondence, averaged clip–sentence matching approximates the unavailable low-level loss and relaxes the requirement that videos and paragraphs contain equal numbers of clips and sentences.The authors note that a more refined soft-assignment approximation is left for future work.
4 Experiments
Experiments evaluate hierarchical cross-modal embeddings on retrieval and downstream tasks, comparing baselines, HSE variants, learning objectives, proposal settings, incomplete observations, and captioning or action-recognition transfer.
- Experimental Setup: The study evaluates video-paragraph retrieval, video captioning, and action recognition using large-scale video datasets.ActivityNet Dense Caption and DiDeMo support the retrieval experiments, with additional downstream evaluations.
- Retrieval Results: Over ∼15% with C3D and ∼30% with InceptionV3, hierarchical models improve previous state-of-the-art Recall@1 results.The comparison is reported across ActivityNet and DiDeMo using both visual feature types.
- Retrieval Results: HSE variants outperform the flat FSE baseline across all retrieval metrics, while layer-wise reconstruction further improves performance over hse[τ=0] on most metrics.The τ ≠ 0 variant uses multiple decoders for layer-wise reconstruction and gives the best experimental results.
- Ablation Studies: Joint learning with low-level and high-level correspondences improves retrieval performance in all evaluated scenarios.With weak correspondence, adding the low-level loss still outperforms models without it and sometimes matches strong-correspondence results.
- Proposal Evaluation: With proposal-based segments, HSE is mostly better than or comparable with FSE, although pretrained action proposals poorly match Dense Caption segments.The authors leave stronger proposal methods for future work.
- Retrieval with Incomplete Video and Paragraph: With fewer than the average number of observed clips and sentences, the model reaches ∼70% of final retrieval performance and already beats prior state of the art with one pair.Retrieval performance increases as more clips and sentences are observed.
- Downstream Tasks: HSE outperforms the baseline on most captioning metrics and supports superior zero-shot action-recognition transfer, but remains below Dense on captioning.The authors attribute Dense’s advantage to specialized context aggregation for caption generation, which the retrieval-optimized embedding model lacks.
5 Conclusion
The paper proposes a cross-modal approach that jointly models videos and texts by leveraging their hierarchical structures. It aligns video–text correspondences across multiple granularities in separate semantic spaces.
- The proposed approach jointly models videos and texts using their intrinsic hierarchical structures.
- The model considers correspondences between videos and texts at multiple granularities.
- Derived loss functions align paired clips and sentences, as well as paired videos and paragraphs, within their respective semantic spaces.
A Implementation Details
The experiments use C3D features under the standard ActivityNet setting and newer Inception V3 features for retrieval on ActivityNet and DiDeMo. Retrieval text inputs use pretrained GloVe word embeddings that are fine-tuned.
- C3D frame-wise features are extracted with temporal stride 16 and reduced to 500 dimensions using PCA.
- Inception V3 penultimate-layer activations pretrained on Kinetics are used for retrieval on ActivityNet and DiDeMo.
- Video frames for Inception V3 are resized to 299 × 299, without test-time data augmentation.
- Retrieval word embeddings are initialized with 300-dimensional GloVe vectors pretrained on 840B web-crawled data and then fine-tuned.
Training Details
The hierarchical embedding models use GRU encoders and decoders, with sequence embeddings obtained by channel-wise max pooling over GRU outputs. Training uses Adam with scheduled learning-rate decay for 15 epochs.
- The low-level encoder processes frame-wise or word-wise representations, and its subsequent embedding initializes the high-level encoder.
- GRU sequence encoders and decoders use hidden dimension 1,024, and sequence embeddings use channel-wise max pooling over GRU outputs.
- Adam training starts at learning rate 0.001 and decays the rate by 10 every 10 epochs.
- All loss margins are set to 0.2, losses are normalized by batch size, and models are trained for 15 epochs on ActivityNet and DiDeMo.
Ablation Study with Different Learning Objectives
The ablation study removes individual learning objectives from HSE on ActivityNet retrieval and uses recall-versus-rank curves to examine the contribution of low-level objectives. Reconstruction and low-match losses are identified as the most useful.
- The study removes one loss at a time from HSE and reports video and paragraph retrieval performance on ActivityNet.
- The reconstruction loss and low-match loss are the most useful objectives in the ablation study.
- Recall-versus-rank curves provide supportive evidence about the effect of low-level objectives.
- Figure 6 compares Video-to-Paragraph and Paragraph-to-Video retrieval for hse[τ=0] and hse on ActivityNet validation set 1.
B.2 Ablation Study on Reconstruction Balance Term
The study selects the reconstruction-loss balance term τ on ActivityNet and evaluates hierarchical modeling against feature-sequence baselines. Results show that tuning τ and combining hierarchical modeling with feature reconstruction improve retrieval performance.
- Balance-term ablation: τ=0.0005 yields near-peak R@1 and R@5 retrieval performance on ActivityNet val2 under the controlled ablation.The experiment uses Inception V3 visual features as input.
- Retrieval performance: hse consistently outperforms fse and hse[τ=0] on ActivityNet val2 video and paragraph retrieval.The comparison reports standard deviations from three randomly seeded experiments.
- Retrieval performance: The comparison attributes the observed improvement to the combined importance of hierarchical modeling and feature reconstruction.hse[τ=0] isolates the no-reconstruction-balance setting, while fse is the feature-sequence baseline.
- Qualitative analysis: Qualitative evaluation covers positive and negative cases for both paragraph retrieval and video retrieval using hse, hse[τ=0], and fse.The examples mark correct samples in green and incorrect samples in red.
- Qualitative analysis: Some hse failures still retrieve items that are very relevant to the query, despite missing the exact correct text or video.The qualitative analysis examines these cases alongside successful retrievals.