Source-linked AI summary
A Joint Sequence Fusion Model for Video Question Answering and Retrieval
Youngjae Yu, Jongseok Kim, Gunhee Kim
TL;DR
Video-language tasks require models that can compare complex frame and word sequences beyond a single-vector representation. JSFusion builds dense pairwise sequence representations and decodes hierarchical matches, achieving the best reported LSMDC performance and outperforming many state-of-the-art methods on MSR-VTT.
Problem
Existing approaches struggle to compare multiple relations between frame and word subsequences, while hierarchical methods require ground-truth parse trees or segmentation labels.
Method
JSFusion combines a Joint Semantic Tensor for dense pairwise frame-word embeddings with a Convolutional Hierarchical Decoder that learns bottom-up attention-based hierarchical matches.
Results
JSFusion achieved the best reported performance on three LSMDC tasks and outperformed many state-of-the-art methods on MSR-VTT video QA and retrieval tasks.
Takeaways & Limitations
The model provides a multimodal sequence matching approach adaptable to video retrieval and video question answering.
Takeaways & Limitations
The demonstrated applications focus on video-language tasks, while broader multimodal applications are proposed as future work.
Abstract
from arXiv · showhide
We present an approach named JSFusion (Joint Sequence Fusion) that can measure semantic similarity between any pairs of multimodal sequence data (e.g. a video clip and a language sentence). Our multimodal matching network consists of two key components. First, the Joint Semantic Tensor composes a dense pairwise representation of two sequence data into a 3D tensor. Then, the Convolutional Hierarchical Decoder computes their similarity score by discovering hidden hierarchical matches between the two sequence modalities. Both modules leverage hierarchical attention mechanisms that learn to promote well-matched representation patterns while prune out misaligned ones in a bottom-up manner. Although the JSFusion is a universal model to be applicable to any multimodal sequence data, this work focuses on video-language tasks including multimodal retrieval and video QA. We evaluate the JSFusion model in three retrieval and VQA tasks in LSMDC, for which our model achieves the best performance reported so far. We also perform multiple-choice and movie retrieval tasks for the MSR-VTT dataset, on which our approach outperforms many state-of-the-art methods.
1 Introduction
JSFusion addresses the difficulty of measuring semantic similarity between video and language sequences by combining dense pairwise representations with hierarchical matching. It is evaluated across video QA and retrieval tasks, achieving leading reported performance on LSMDC and outperforming many state-of-the-art methods on MSR-VTT.
- Motivation: Video-language understanding requires hidden joint representations that relate word sequences to frame sequences and multiple latent video events.The introduction contrasts this need with video classification approaches that assign only a few labels to a video.
- Motivation: Hierarchical matching is better suited to comparing subevents in videos with short phrases than direct comparison of entire sequence modalities.Earlier hierarchical approaches required ground-truth parse trees or segmentation labels.
- Approach: JSFusion combines the Joint Semantic Tensor, which encodes pairwise frame-word embeddings in a 3D tensor, with the Convolutional Hierarchical Decoder, which learns hierarchical matches.Attention mechanisms refine the tensor, promote well-matched patterns, prune misaligned ones, and produce a semantic matching score.
- Evaluation: The model was evaluated on three LSMDC tasks and additional MSR-VTT video QA and retrieval benchmarks.The LSMDC tasks include multiple-choice, movie retrieval, and fill-in-the-blank settings.
- Results: JSFusion achieved the best reported performance on the LSMDC tasks and outperformed many state-of-the-art methods on MSR-VTT.The MSR-VTT comparisons cover diverse video topics, including TV shows, web videos, and cartoons.
2 Related Work
The paper situates its work at the intersection of video retrieval and video question answering, both of which require modeling relationships between visual sequences and language. It distinguishes JSFusion from prior approaches through dense pairwise sequence embeddings and bottom-up hierarchical matching.
- Video-language research in this context centers on video retrieval and video question answering.
- Video retrieval with natural language sentences: Video retrieval is challenging because models must learn latent relations between frame sequences and descriptive word sequences rather than simply aggregate images.
- Video retrieval with natural language sentences: JSFusion first forms dense pairwise embeddings, then composes higher-level similarity matches bottom-up with hierarchical attention, improving robustness to local subset matching.
- Video question answering: Video question answering is harder than image-based VQA because it requires spatio-temporal reasoning and large-scale annotated data.
- Video question answering: Crowdsourced datasets have extended video QA across movies and animated GIFs, including LSMDC tasks such as multiple-choice and fill-in-the-blank.
- Video question answering: In multiple-choice video QA, systems select one answer from five descriptive sentences using ranking losses, joint-space nearest-neighbor search, or temporal attention.
3 The Joint Sequence Fusion Model
JSFusion matches multimodal sequences by first forming a densely aligned Joint Semantic Tensor, then decoding hierarchical compatibility patterns with gated convolutions. Its preprocessing and encoders represent words and video frames with bounded sequence lengths and contextual features before attention-based fusion.
- 3.1 Preprocessing: Words use pretrained 300-dimensional GloVe embeddings, while sentences are capped at 40 words; excess words are discarded because only 0.07% exceed the limit.The vocabulary includes words occurring more than three times; for LSMDC, its size is |V| = 16,824.
- 3.1 Preprocessing: Videos are sampled at five frames per second and represented by concatenated ResNet-152 visual features and VGGish audio features, with at most 40 frames retained.Longer videos use 40 equidistant frames, and larger frame limits show no performance gain.
- 3.2 The Joint Semantic Tensor: JST forms pairwise frame-word representations in a 3D tensor and applies learned sigmoid attention to emphasize aligned pairs while pruning unmatched joint representations.The joint representation is j_nm = α_nmγ_nm, with α_nm computed from the pairwise tensor and γ_nm produced by dense layers.
- 3.2 The Joint Semantic Tensor: Bidirectional LSTMs encode word sequences and a one-dimensional CNN encodes video frames, producing contextual modality-specific representations for fusion.Each modality representation concatenates encoder states with its input features before separate dense projections.
- 3.3 The Convolutional Hierarchical Decoder: CHD recursively processes the JST tensor through convolutional and convolution-gating layers that activate positively aligned patterns and suppress misaligned ones.Three convolutional and gating stages are applied, followed by mean pooling and dense layers to produce a single similarity score.
- 3.3 The Convolutional Hierarchical Decoder: Attention visualizations show darker weights for better word-frame or structural matches, while gating produces low activations for negative sequence pairs.The model is motivated by the difficulty of comparing long videos and sentences in a single vector space and instead uses dense pairwise and multistage matching.
4 Experiments
JSFusion is evaluated across three LSMDC video-language tasks and two newly annotated MSR-VTT tasks, with comparisons against published, reproduced, and implemented baselines. It achieves the best reported performance across both datasets, while qualitative and ablation analyses examine its matching behavior and component contributions.
- Datasets and tasks: The experiments cover LSMDC movie retrieval, multiple-choice, and fill-in-the-blank, plus newly created MSR-VTT multiple-choice and retrieval tasks.The LSMDC benchmark includes 118,114 sentences and 118,081 short video clips; the MSR-VTT annotations follow the LSMDC task protocols.
- Evaluation: The retrieval evaluation reports Recall@1, Recall@5, Recall@10, and Median Rank, while multiple-choice and fill-in-the-blank use accuracy.Recall@k measures the percentage of ground-truth videos in the first k retrieved results, whereas Median Rank is lower-is-better.
- Quantitative results: JSFusion achieves the best performance with significant margins over all baselines across the three tasks on both datasets.The comparison includes multimodal embedding, spatio-temporal attention, and language-model-based QA methods; a single JSFusion model also surpasses the runner-up ensemble on fill-in-the-blank.
- Ablation study: Replacing the proposed components with simpler fusion, pooling, or encoding variants lowers performance, indicating that CHD and the full JSFusion design contribute across tasks.The simple LSTM-fusion variant performs significantly worse, and comparisons with max/mean pooling and alternative encoders identify CHD as critical.
- Ablation study: Removing CHD attention causes MC accuracy drops of 4.1%p on LSMDC and 4.2%p on MSR-VTT, while retrieval R@10 drops by 5.7%p and 3.7%p, respectively.The no-audio variant remains stronger than other baselines but is slightly worse than the original model.
- Qualitative results: Qualitative examples show successful matching of behaviors, visual concepts, sentence structure, and phrase meaning, alongside failures on subtle motions, details, subsequence selection, and visual distinctions.Examples include retrieving videos containing a woman’s face instead of a small facial motion and choosing framed picture rather than flash picture.
5 Conclusion
JSFusion measures hierarchical semantic similarity between multimodal sequences through JST and CHD, improving video-language understanding. It performs best on LSMDC challenge tracks and outperforms many state-of-the-art models on MSR-VTT VQA and retrieval tasks, while broader modalities remain future work.
- JSFusion measures hierarchical semantic similarity between two multimodal sequence data using the Joint Semantic Tensor and Convolutional Hierarchical Decoder.The model is designed for multimodal matching and video question answering.
- JSFusion significantly improves video understanding through natural-language descriptions across video-language tasks.
- JSFusion achieves the best performance in LSMDC challenge tracks and outperforms many state-of-the-art models on MSR-VTT VQA and retrieval tasks.
- The authors plan to extend JSFusion to multimodal retrieval involving videos-to-voices and text-to-human motions.These modalities are identified as directions for future exploration.