Source-linked AI summary
Towards Unified Vision-Language Models with Incomplete Multi-Modal Inputs
Xiang Fang, Wanlong Fang, Changshuo Wang, Keke Tang, Daizong Liu, Siyi Wang, Wei Ji
TL;DR
VLMs typically assume complete video-text inputs, although real-world applications can produce incomplete modalities and expose limitations in existing alignment methods. The paper introduces a unified completeness network for incomplete video-language alignment, and experiments show it improves state-of-the-art methods across multiple multimodal tasks as a plug-and-play module.
Problem
Existing VLM-based methods rely on complete video-text pairs, while real-world missing frames and words create incomplete inputs that these methods cannot effectively align.
Method
The paper proposes a unified completeness network combining multimodal feature approximation, multimodal knowledge distillation, and multi-granularity multimodal integration for incomplete video-language inputs.
Results
The framework improves state-of-the-art VLM-based methods across video-text retrieval, video question answering, and video sentence grounding as a plug-and-play module.
Takeaways & Limitations
A single completeness framework can be applied across multiple incomplete video-language tasks and existing task-specific models.
Abstract
from arXiv · showhide
Video-Language Models (VLMs) have demonstrated impressive multi-modal reasoning capabilities across diverse computer vision applications. However, these VLMs are task-specific and assume that both video and language inputs are complete. However, real-world VLM applications might face challenges due to deactivated sensors (e.g., cameras are unavailable due to data privacy), yielding modality-incomplete data and leading to inconsistency between training and testing data. While straightforward incomplete input can boast training generalization-ability and lead to training failure, its potential risks to VLMs regarding safety and trustworthiness have been largely neglected. To this end, we make the first attempt to propose a unified incomplete video-language model to process the incomplete multi-modal inputs. Extensive experimental results show that our method can serve as a plug-and-play module for previous works to improve their performance in various multi-modal tasks.
Introduction
VLMs perform well on complete video-language tasks but depend heavily on complete paired inputs, limiting their reliability when real-world data lose frames or words. The paper introduces an incomplete video-language setting and a unified completeness network with feature approximation, knowledge distillation, and multi-granularity integration.
- VLMs support downstream applications including text-to-video retrieval and video question answering.
- Network instability and data loss can produce incomplete multimodal inputs, including videos with missing frames and texts with missing words.
- Existing alignment methods rely on complete video-text pairs during training and inference, while missing frames prevent correct motion reconstruction and can degrade performance or cause model failure.
- The paper defines incomplete video-text alignment for unbalanced incomplete inputs, where video and text have different incompleteness rates.
- The unified completeness network combines multimodal feature approximation, multimodal knowledge distillation, and multi-granularity multimodal integration.
- The method is presented as a plug-and-play module that improves task-specific methods across video-text retrieval, video question answering, and video sentence grounding.
Related Works
Research on incomplete multimodal inputs addresses modality incompleteness across multiple tasks using strategies such as improved fusion and data augmentation.
- Prior work improves robustness to modality-incomplete data across various multimodal tasks.
- Existing approaches include optimizing multimodal fusion strategies and conducting data augmentation.
Our Proposed Method
The method is a unified pipeline for incomplete video-text pairs that approximates missing features, distills knowledge from complete inputs, and integrates modalities at multiple granularities. It is designed as a plug-and-play approach for downstream multi-modal tasks.
- Pipeline: The pipeline encodes incomplete videos frame-by-frame and texts word-by-word before completing missing modality features.Missing frames or words are represented as null entries, and feature encoders extract the available visual and textual representations.
- Multi-modal Feature Approximation: The Multi-modal Feature Approximation module uses cross-modal and intra-modal nearest neighbors to estimate missing frame and word features.K-reciprocal neighbors and Jaccard distances identify semantically relevant neighbors, whose features are combined through affinity matrices.
- Multi-modal Feature Approximation: A pre-trained large multi-modal model refines the approximated video and text memories after graph-based feature completion.The approximation constructs graph relationships that transmit information among samples, then applies the large multi-modal model to refine the resulting memories.
- Multi-modal Knowledge Distillation: The Multi-modal Knowledge Distillation module trains a student on incomplete pairs against a teacher trained on complete video-text pairs.This addresses the computational cost of completing every missing frame and word while reducing reliance on the complete modality.
- Multi-granularity Multi-modal Integration: The Multi-granularity Multi-modal Integration module uses shared prototypes and matching-probability weights to align video-text pairs at task-appropriate granularities.Prototype cross-attention reconstructs modality features, while noun-phrase overlap weights support fine-grained alignment for tasks requiring instance-level correspondence.
Experiment
Experiments evaluate the framework across retrieval, question answering, and grounding tasks under incomplete video-text inputs. The framework improves prior methods as a plug-and-play module, while ablations and efficiency comparisons examine its components and practical performance.
- Experiment: The evaluation covers video-text retrieval, video sentence grounding, and VideoQA using open-source datasets, with incomplete pairs generally defined by a 30% missing rate.Metrics include retrieval recall, grounding at specified IoU thresholds, and VideoQA categories.
- Performance Comparison: Incomplete video-text inputs cause substantial degradation for prior retrieval and VideoQA methods because missing frames and words are directly concatenated or insufficiently understood.The reported comparisons include text-to-video and video-to-text retrieval, as well as VideoQA on incomplete pairs.
- Performance Comparison: The framework improves state-of-the-art VSG methods across all reported metrics when used as a plug-and-play module.The comparison includes VSG results summarized in Table 4, Table 5, and Figure 4.
- Performance Comparison: Visualization results show poor performance from state-of-the-art methods on incomplete multi-modal datasets across three downstream tasks.Figure 5 presents visualizations for different downstream tasks and incomplete datasets.
- Efficiency Comparison: The framework achieves faster processing speeds with relatively fewer learnable parameters than most compared methods in the VSG efficiency comparison.The efficiency comparison uses 2D-TAN as the base model and is reported for VSG on TACoS.
- Ablation Study: Ablation studies remove individual components, including multi-modal feature approximation, to evaluate their contributions to the VSG framework.The ablation study uses G2L as the base model and investigates key modules individually.
Conclusion
The paper targets incomplete video-text alignment and proposes a unified completeness network for incomplete multi-modal inputs. Experiments across retrieval, VideoQA, and grounding show that the framework can improve state-of-the-art VLM-based methods as a plug-and-play module.
- Conclusion: The paper proposes incomplete video-text alignment and a unified completeness network for modality-incomplete challenges across downstream multi-modal tasks.The targeted tasks include video-text retrieval, video question answering, and video sentence grounding.
- Conclusion: Extensive experiments on incomplete multi-modal datasets show that the framework improves state-of-the-art VLM-based works as a plug-and-play module.The reported scope spans multiple downstream multi-modal tasks.