PaperlayerView exact revision on arXiv

Source-linked AI summary

Valley: Video Assistant with Large Language model Enhanced abilitY

Ruipu Luo, Ziwang Zhao, Min Yang, Zheming Yang, Minghui Qiu, Tao Wang, Zhongyu Wei, Yanhao Wang, Cen Chen

arXiv:2306.07207v3cs.CVcs.AIcs.CL

TL;DR

Valley addresses the limited exploration of joint video-language understanding by combining new alignment and instruction datasets with temporal modeling and staged training. It shows strong video-understanding performance, including state-of-the-art zero-shot results on several question-answering benchmarks and broad Video-Bench superiority, while its v1 temporal structure has a stated time-dimension limitation.

  • Problem

    Joint video-language understanding by conversational LLM assistants remains insufficiently explored, while existing video datasets and models provide limited quality, diversity, task coverage, and temporal modeling.

  • Method

    Valley combines Valley-702k and Valley-instruct-73k with a ViT-L/14 vision encoder, three temporal-modeling designs, and two-stage training for video comprehension and instruction following.

  • Results

    Valley achieves state-of-the-art zero-shot performance on MSVD, MSRVTT, and ActivityNet question-answering, and outperforms all competitors across Video-Bench’s three tasks.

  • Takeaways & Limitations

    Valley demonstrates potential as an effective video assistant for video, image, and language understanding and generation across multimodal tasks.

  • Takeaways & Limitations

    Valley-v1’s temporal structure can confuse the time dimension because features from different timestamps are not treated as equally important.

Abstract

from arXiv · show

Large Language Models (LLMs), with remarkable conversational capability, have emerged as AI assistants that can handle both visual and textual modalities. However, their effectiveness in joint video and language understanding has not been extensively explored. In the paper, we introduce Valley, a multi-modal foundation model that is designed to enable enhanced video comprehension and instruction-following capabilities. To this end, we construct two datasets, namely Valley-702k and Valley-instruct-73k, to cover a diverse range of video-text alignment and video-based instruction tasks, such as multi-shot captions, long video descriptions, action recognition, causal inference, etc. Then, we adopt ViT-L/14 as the vision encoder and explore three different temporal modeling modules to learn multifaceted features for enhanced video understanding. In addition, we implement a two-phase training approach for Valley: the first phase focuses solely on training the projection module to facilitate the LLM's capacity to understand visual input, and the second phase jointly trains the projection module and the LLM to improve their instruction following ability. Extensive experiments demonstrate that Valley has the potential to serve as an effective video assistant, simplifying complex video-understanding scenarios. Our code and data are published anonymously at https://github.com/valley-vl/Valley.

1 INTRODUCTION

Video-language assistants build on LLMs’ conversational and instruction-following strengths, but joint video understanding remains underexplored and constrained by limited datasets and temporal modeling. Valley addresses these gaps with new video datasets, temporal modules, and staged training, achieving strong benchmark results.

  • Motivation: Joint video-language understanding remains underexplored despite LLMs’ success in conversational and multimodal assistance.Existing multimodal assistants have progressed especially in image understanding, supported by abundant image-text data.
  • Challenges: Existing video datasets suffer from poor quality, limited content diversity, insufficient task variety, and inadequate temporal-feature modeling.These shortcomings make high-quality video-text and instruction data difficult to construct.
  • Approach: Valley combines about 100,000 collected videos, Valley-702k, Valley-instruct-73k, three temporal-modeling strategies, and two-stage training.The datasets cover alignment and instruction tasks including multi-shot captions, long descriptions, action recognition, and causal inference.
  • Results: Valley achieves state-of-the-art zero-shot performance on MSVD, MSRVTT, and ActivityNet video question-answering tasks.It also provides leading results for video-based text generation.
  • Results: Valley outperforms all competing methods across Video-Bench’s Video-Exclusive, Prior-knowledge, and Decision-Making tasks.The experiments also demonstrate chain-of-thought and few-shot capabilities.

2 RELATED WORK

Related work establishes LLMs as capable language assistants and surveys multimodal systems that connect LLMs with visual models. Video instruction datasets remain vulnerable to hallucinations and information gaps caused by inadequate captions.

  • Large Language Models: LLMs support language understanding, reasoning, few-shot and zero-shot prompting, and assistant applications such as Alpaca, Vicuna, and ChatGLM.Their large-scale pretraining enables broad task-transfer capabilities.
  • Video Instruction Data: Figure 2 contrasts hallucinations in instruction data generated from dense captions by Video-ChatGPT and short captions by VideoChat.The illusion portions are highlighted in red.
  • Multimodal LLMs: Multimodal LLM methods either use an LLM to schedule existing multimodal models or train a multimodal model based on an LLM.The surveyed approaches organize or align multimodal inputs with language models.
  • Video Instruction Data: Figure 3 presents an example of instruction data generated from a collected detailed caption.The construction process uses detailed captions as input for instruction generation.

3 DATASET CONSTRUCTION

Valley’s datasets combine cleaned web video-text pairs with detailed captions and ChatGPT-generated instruction data. The resulting collection spans diverse video tasks and includes 73k instruction examples across dialogue, reasoning, and description formats.

  • Existing Data Limitations: VideoChat and Video-ChatGPT datasets are limited by short WebVid captions, activity-focused ActivityNet captions, and insufficient information for high-quality instruction generation.Dense captioning methods can introduce hallucinations, motivating improved data construction.
  • Dataset Design: Valley constructs Valley-702k for visual-text alignment and Valley-instruct-73k for improving instruction-following capabilities.WebVid2M is filtered using phrase extraction and frequency thresholds to improve the alignment data.
  • Video Collection: The collected videos average 40 seconds, with some exceeding 5 minutes, providing substantial temporal context for detailed descriptions.Figure 4 summarizes description length, duration, and category distributions for about 100k JukinMedia videos.
  • Instruction Data: Valley-instruct-73k contains 37k dialogue pairs, 26k complex-reasoning question-answer pairs, and 10k detailed-description instruction pairs.ChatGPT generates the instruction data from manually written examples using few-shot prompts.

4 THE VALLEY MODEL

Valley combines a vision encoder, temporal modeling module, projection layer, and LLM to process videos of varying lengths alongside text. Its temporal module aggregates frame-level spatial features through three alternative structures that use average pooling, learned temporal weighting, or temporal variation modeling.

  • Architecture: Valley incorporates a temporal modeling module into the vision encoder to aggregate grid features from video frames into unified vision tokens.The architecture is designed to let an LLM jointly process videos of varying lengths and individual images.
  • Architecture: Videos are sampled at 0.5 FPS, and ViT-L/14 extracts 256 spatial patch features plus one global [CLS] feature per frame.The sampled frame features provide the inputs to temporal aggregation.
  • Architecture: The model concatenates temporally modeled patch features with global features from all frames to preserve global temporal information.The concatenated representation forms the video representation ZV, where ⊕ denotes concatenation.
  • Architecture: A projection layer maps video patch and global features into the language embedding space before the projected visual features and text embeddings enter the LLM.The LLM then generates a response from the combined visual and textual inputs.
  • Temporal modeling module: The v1 temporal structure averages spatial-token representations across frames but does not account for unequal importance across timestamps.This can cause confusion in the time dimension because different timestamps may carry different importance.
  • Temporal modeling module: The v2 structure learns frame-importance scores for weighted averaging, while v3 adds a transformer-derived temporal-variation feature to average-pooled features.Both designs extend the basic averaging strategy with explicit temporal modeling.

5 EXPERIMENTS

Valley is evaluated across diverse video, image, and textual understanding tasks using zero- and few-shot settings. Its strongest results appear in zero-shot video question answering, video generation, Video-Bench, image metaphor understanding, and selected ScienceQA settings.

  • 5.1 SETUP: Valley uses a two-stage training scheme that first pre-trains the projection layer and then jointly fine-tunes the language model and projection layer.The pre-training phase uses image-text and video-text pairs, while instruction tuning uses combined video- and image-based instruction data.
  • 5.1 SETUP: Valley is evaluated on six datasets spanning video understanding, image understanding, and textual tasks in zero- and few-shot settings.The video datasets are MSVD-QA, MSRVTT-QA, ActivityNet-QA, and Video-ChatGPT; the image datasets are ScienceQA and MemeCap.
  • 5.2 EXPERIMENTAL RESULTS: Valley-v3 is the top-performing variant across the three zero-shot video question-answering datasets, outperforming most baseline models.The paper attributes this performance to Valley-v3’s ability to understand video context and provide more accurate and reasonable answers.
  • 5.2 EXPERIMENTAL RESULTS: Valley-v3 achieves the best performance in four of five Video-ChatGPT text-generation aspects: correctness, contextual understanding, temporal understanding, and consistency.Valley is slightly inferior to Video-ChatGPT in detail orientation, which the paper relates to missing detailed object descriptions in its instruction data.
  • 5.2 EXPERIMENTAL RESULTS: Valley outperforms all competing methods in Video-Bench’s Video-Exclusive, Prior-knowledge, and Decision-Making tasks.This result is reported as evidence of Valley’s effectiveness on video-based tasks.
  • 5.2 EXPERIMENTAL RESULTS: Valley’s generated descriptions are semantically closer to MemeCap ground truth than outputs from MiniGPT4 and Flamingo, while chain-of-thought ScienceQA performance is generally comparable to GPT-3.5.Valley also shows a certain one-shot inference ability and exceeds GPT-3.5 in some ScienceQA classes, including G7-12, SOC, and TXT.

6 CONCLUSION

Valley is presented as a unified multimodal foundation model for video, image, and language understanding. The paper combines cleaned and newly collected data with temporal modeling and two-stage training, and reports effectiveness across multimodal understanding and generation tasks.

  • 6 CONCLUSION: Valley unifies video, image, and language understanding within a multimodal foundation model.The model is trained using video-text alignment and video instruction datasets constructed with assistance from ChatGPT.
  • 6 CONCLUSION: The final training resources include the Valley-702k and Valley-instruct-73k datasets, with cleaned WebVid video-text pairs used to improve data quality.The instruction dataset is constructed as a high-quality, multi-task video instruction-following resource.
  • 6 CONCLUSION: Valley uses three temporal-modeling structures and a two-stage training procedure.The conclusion identifies these design choices alongside dataset construction as core components of the model.
  • 6 CONCLUSION: Experiments on several multimodal datasets confirm Valley’s effectiveness in video, image, and textual understanding and generation.The stated future goal is a more intuitive and personalized multimodal assistant for human-machine interactions.

A IMPLEMENTATION DETAILS OF VALLEY

The appendix reports Valley’s implementation settings and two-stage optimization procedure. Stage 1 aligns visual and language representations, while Stage 2 fine-tunes the connected language components for instruction following.

  • A IMPLEMENTATION DETAILS OF VALLEY: Valley’s implementation settings are summarized in Table 6.
  • A IMPLEMENTATION DETAILS OF VALLEY: Stage 1 pre-trains the projection layer while using Open-CLIP-L-224 for the ViT, Stable-Vicuna-13b for the LLM, and random initialization for the connection module.The connection module bridges the visual and language components.
  • A IMPLEMENTATION DETAILS OF VALLEY: Stage 2 initializes the LLM and connection module from Stage 1 and jointly fine-tunes them with a reduced learning rate of 5 × 10^-5.AdamW, cosine decay, and bfloat16 precision remain unchanged from pre-training.

B.1 EXAMPLE OF TRAINING DATA IN ‘Valley-702k’

The training-data example pairs a human request for concise video description with a generated caption. The example depicts a girl using a computer in an indoor setting.

  • B.1 EXAMPLE OF TRAINING DATA IN ‘Valley-702k’: The example uses the human instruction “Describe the following video concisely” followed by a video placeholder.
  • B.1 EXAMPLE OF TRAINING DATA IN ‘Valley-702k’: The corresponding caption describes a smiling young girl interacting with a computer keyboard while seated near a white brick wall.

B.2 EXAMPLE OF TRAINING DATA IN ‘Valley-instruct-73k’

The training data includes video-grounded question-and-answer conversations, illustrated by a question about performers executing a flipping trick at an airport and a generated response.

  • The example demonstrates the conversational format used for video-based instruction data.
  • A sample asks how performers execute a flipping trick at an airport.
  • The generated answer explains the trick using acrobatic skills and available airport space, beginning with an open area and a running start.

C CASE STUDIES

The case studies show Valley producing video-based responses that describe relationships, capture emotional tone, add imaginative interpretations, and make detailed observations. Additional examples cover image metaphors and the construction and evaluation prompts used for instruction data and question answering.

  • Video case studies: The case studies present Valley as having video understanding and instruction-following capabilities.The paper uses several practical cases to demonstrate these capabilities.
  • Video case studies: Valley accurately describes relationships and emotional tone in videos involving cats, people, and dogs.The examples identify interacting objects, emphasize bonds and mutual enjoyment, and capture playful or funny aspects.
  • Video case studies: Valley generates imaginative and engaging interpretations, including a short story and a cooking-action description.These outputs add creative narrative content while remaining grounded in the illustrated video cases.
  • Video case studies: Valley provides detailed observations about actions and corrects errors in a supplied video description.The examples include a dog wearing clothes and interacting with a computer, as well as correction of an erroneous cooking description.
  • Image metaphor case studies: Figure 9 presents two examples of Valley understanding image metaphors by comparing human-annotated and model-generated image interpretations.The figure distinguishes the human understanding with a blue box and Valley’s understanding with a green box.
  • Prompts and evaluation: The instruction-data examples use few-shot prompts for detailed descriptions, conversations, and complex reasoning, while a prompt supports question-answering evaluation.The paper states that manually written examples guide ChatGPT in generating detailed-description and complex-reasoning data, and that evaluation uses answer accuracy and consistency.
Loading 2306.07207v3…