VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding

1 Introduction

• Image-only approaches can lack explicit temporal context, while video-only approaches are constrained by sparse, lower-resolution processing.The paper also notes that concatenating image embeddings without pooling can increase context length and computational complexity.Source: §1 · p. 1
• VideoGPT+ combines an image encoder for rich spatial detail with a video encoder for global temporal context and uses segment-wise sampling to capture information across the video.Unlike uniform sampling, segment-wise sampling divides the video into smaller segments to capture representative information from different portions.Source: §1 · p. 2
• A visual adapter projects and pools image and video features into a common space while reducing computational complexity.The aligned features let the model use combined spatial and temporal information.Source: §1 · p. 2
• VideoGPT+ outperforms previous state-of-the-art approaches across VCGBench, MVBench, and zero-shot question-answering benchmarks.The comparison is shown across multiple benchmarks in Figure 1.Source: §1 · p. 2, Figure 1 · p. 2
• The authors develop VCG+ 112K, a semi-automatically annotated instruction dataset, and propose VCGBench-Diverse for evaluation across 18 video domains.VCG+ 112K provides dense captions and spatial-understanding and reasoning QA pairs; VCGBench-Diverse extends evaluation to diverse domains.Source: §1 · p. 2

Video Conversation Models

• Video-ChatGPT [1] uses spatial and temporal pooling, while other approaches reduce visual tokens through clustering, query-based attention, or lightweight CNNs [2] [5] [19] [20].The paper describes Video-ChatGPT [1] and Video-LLaMA [15] as extending image-based models with temporal encoding, but notes that frame-level features can quickly fill the LLM context window.Source: §2 · pp. 2–3
• VideoGPT+ combines image features for spatial detail with video features for temporal context, using segment-wise sampling to capture information across video segments.The overview depicts the features being projected, pooled, concatenated, and passed to an LLM.Source: §2 · p. 3, Figure 2 · p. 3

Video Instruction Tuning Datasets

• VideoChat [5], Video-ChatGPT [1], and VideoChat2 [7] provide instruction datasets of different sizes and compositions; the authors add VCG+ 112K with dense captions and QA targeting spatial, temporal, and reasoning understanding.VideoChat uses 7K instructions, Video-ChatGPT introduces VideoInstruct100K, and VideoChat2 combines existing datasets into a 1.9M joint image-video dataset.Source: §2 · p. 3

Video Conversation Benchmarks

• VCGBench [1] draws only on ActivityNet videos, while MVBench [7] mostly contains short videos; VCGBench-Diverse adds 4,354 QA pairs from 877 videos across 18 domains.The new benchmark is intended to address limited diversity and generalization coverage.Source: §2 · p. 3

Overall Architecture

• The architecture samples video segments, encodes them with image and video encoders, projects both feature sets into language space, adaptively pools and concatenates the tokens, then feeds them to an LLM.The LLM is fine-tuned with LoRA [31] using autoregressive next-token prediction.Source: §3 · p. 4

Segment-wise Sampling

• Segment-wise sampling divides a video into K segments, each with n = T/K frames, to capture fine-grained temporal cues.The video segments are downsampled for video encoding; the approach is presented as a manageable alternative to uniform sampling under video-encoder constraints.Source: §3 · p. 4

Dual Vision Encoder

• The image encoder produces local frame-level features, while the video encoder processes low-resolution segments to produce global segment-level features.The selected pretrained encoders are CLIP ViT-L/14 (336 × 336) [29] and InternVideo-v2 (224 × 224) [30].Source: §3 · p. 4

Visual Adapter

• Separate trainable MLP projection layers map frozen image and video encoder features into language space, where 2 × 2 adaptive pooling reduces token length by a factor of four.The pooling is applied over the spatial dimensions of local and global features.Source: §3 · p. 4

Large Language Model

• The LLM input concatenates image embeddings with K segment-wise video embeddings, placing detailed spatial representation across segments alongside global temporal context.The resulting visual embeddings are combined with the user text query before processing by the LLM.Source: §3 · p. 4

VCG+ 112K

• VCG+ 112K improves the Video-ChatGPT [1] VideoInstruct100K annotation process, whose semi-automatic pipeline generates 75K instruction-tuning QA pairs, by improving pair accuracy and quality through keyframe extraction, SoTA large multimodal models for detailed descriptions, and refined instruction generation.Source: §4 · pp. 4–5

Keyframe Extraction

• Selecting one keyframe per extracted scene adapts visual coverage to video content, aiming to reduce insufficient or redundant information from fixed keyframe counts.The paper says this yields more detailed information for rich videos and less redundancy for videos with less content.Source: §4 · p. 5

Frame-Level Descriptions

• LLaVA v1.6 generates dense keyframe descriptions that add spatial attributes, scene context, and object characteristics to concise ground-truth captions.The frame-level descriptions are detailed but noisy and are used to augment the captions.Source: §4 · p. 5

Detailed Video Descriptions

• GPT-4 reconciles frame-level descriptions with ground-truth captions, removes inconsistencies, and creates detailed descriptions with event timelines, actions, object attributes, and scene settings.The resulting structured descriptions are then used to generate QA pairs with GPT-3.5.Source: §4 · p. 5, Figure 3 · p. 5

Improved Instruction Tuning Data

• GPT-3.5 generates descriptive and concise QA pairs covering dense captioning, temporal information, spatial reasoning, event reasoning, and short temporal questions.The figure summarizes the pipeline from captions and frame descriptions through detailed descriptions to instruction-tuning QA pairs.Source: §4 · pp. 5–6, Figure 3 · p. 5

VCGBench-Diverse

• VCGBench-Diverse evaluates video LMMs with 4,354 QA pairs across 18 broad video categories, testing captioning, spatial and temporal understanding, and reasoning.The benchmark also covers five video-capturing methods and six reasoning complexities.Source: Figure 4 · p. 6

5 Proposed Benchmark

• VCGBench-Diverse evaluates video LMM generalization with 877 videos, 18 broad video categories, and 4,354 QA pairs; VCG-Bench [1], by contrast, uses only ActivityNet200 videos.Source: §5 · p. 6
• Its 18 domains range from lifestyle and science to sports, gaming, surveillance, and traffic.The benchmark also varies capture methods and assesses robustness across filming techniques, camera movements, quality levels, and lighting.Source: §5 · p. 6
• Six reasoning complexities include sequential, predictive, causal, contextual, narrative, and analytical reasoning.These dimensions accompany content diversity and varied video-capture methods.Source: §5 · p. 6
• Human annotators describe videos using audio and visual elements; meta-review and random quality checks precede GPT-3.5 QA-pair generation.Videos come from HDVILA, MPII, YouCook2, UCF Crime, and STUD Traffic.Source: §5 · p. 6
• Evaluation retains five VCG-Bench aspects and separately reports dense captioning, spatial understanding, and reasoning.The five aspects are correctness of information, detail orientation, contextual understanding, temporal understanding, and consistency.Source: §5 · pp. 6–7

Experimental Setup

• Evaluation covers VCGBench, VCGBench-Diverse, MVBench, and zero-shot QA; sampling uses 16 frames except 8 for MVBench.Training combines approximately 260K single-turn conversations for VCGBench and VCGBench-Diverse, and approximately 330K for MVBench.Source: §6 · p. 7

VCGBench-Diverse

• 2.47 average score surpasses previous methods on VCGBench-Diverse; spatial and temporal understanding exceed previous best by 0.37 (7.4%) and 0.23 (4.6%), respectively.The benchmark contains 4,354 QA pairs from 877 videos; VideoGPT+ scores 1.38 on dense captioning, 2.80 on spatial understanding, and 3.63 on visual reasoning.Source: §6 · p. 7, Table 2 · p. 8

MVBench

• 7.6% over previous best VideoChat2; VideoGPT+ averages 58.7% across MVBench’s 20 tasks and achieves state-of-the-art results in 14.It places second in four tasks; reported gains include Action Prediction (+12.5%), Object Existence (+27.5%), Moving Direction (+17%), Moving Count (+29%), and Moving Attributes (+32%).Source: §6 · p. 8, Table 3 · p. 8

Zero-shot Question-Answering

• VideoGPT+ achieves superior performance to previous methods across four zero-shot QA datasets.The evaluation covers MSVD-QA, MSRVTT-QA, TGIF-QA, and ActivityNet-QA, with none of the videos included in training.Source: §6 · p. 8, Table 4 · p. 8

Ablations

• 3.28 on VCGBench with dual encoders, versus 3.17 image-only and 3.20 video-only; 2×2 image adaptive pooling and spatial video pooling outperform alternatives.The 4×4 image pooling and temporal video pooling alternatives perform worse.Source: §6 · p. 9, Table 5 · p. 9

VCG+ 112K

• VCG+ 112K improves performance, particularly in detail orientation and temporal understanding.The semi-automatic annotation pipeline produces dense video captions and spatial-understanding and reasoning-based QA pairs.Source: §6 · p. 9

LLM Type

• Better LLMs, including Vicuna 13B and LLaMA-3 8B, bring slight improvements in VCGBench scores.The experiments compare Vicuna 7B and 13B with LLaMA-3 8B.Source: §6 · p. 9, Table 6 · p. 9

7 Conclusion

• VideoGPT+ combines image and video encoders, visual adapters, and segment-wise sampling for video understanding.The conclusion attributes improved benchmark performance to the dual-encoder design, adapters mapping features to a common space, and sampling that retains fine-grained temporal information.Source: §7 · p. 9
• VCG+ 112K provides further gains, and VCGBench-Diverse covers 18 video categories for evaluating video LMMs.The instruction set uses a resource-efficient semi-automated annotation pipeline.Source: §7 · p. 9
• Precise action localization, very long videos, and long paths remain challenges for video LMMs.The conclusion identifies these as areas where major improvements could unlock new applications.Source: §7 · p. 9
• In the sunscreen advertisement example, VideoGPT+ identifies multiple events, reasons about an advertisement, and reads the SPF value and brand name.The qualitative comparison describes these as examples of temporal, reasoning, and spatial understanding.Source: Figure 5 · p. 10

8 Qualitative Results

• VideoGPT+ describes the sunscreen video’s sequence of scene changes, while VideoChat2 struggles to capture all events accurately.The sequence includes sunscreen application, key ingredients, and the product cover.Source: §8 · p. 10
• VideoGPT+ generates an advertisement highlighting sunscreen’s moisturizer function and correctly identifies the SPF value and brand; VideoChat2 struggles with the brand.The example illustrates creative reasoning alongside product-detail identification.Source: §8 · p. 10

• All experiments use 8xA100 40GB GPUs.Source: §9 · p. 11
• VCGBench training takes around 12 hours, while MVBench training takes around 10 hours.Source: §9 · p. 11
• The VCGBench-trained model is used for VCGBench-Diverse and zero-shot question-answering evaluation, and the authors say they will publicly release code, pretrained models, and the dataset.Source: §9 · p. 11

10 Additional Ablations

• The ablation compares interleaving image and video features within each segment against placing all image features before all video features.Both strategies extract image and video features for each video segment before input to the LLM.Source: §10 · p. 11
• The authors propose that different visual adapters make interleaving modalities create a larger distribution shift, hindering learning.Source: §10 · p. 11
• Sequential feature concatenation performs better than interleaved concatenation.The authors attribute this to interleaving features processed by different visual adapters creating a larger distribution shift that hinders learning.Source: Table 7 · p. 11, §10 · p. 11

11 GPT Prompts

• The section covers GPT prompts for dense video descriptions, VCG+ 112K question-answer pairs, and VCGBench-Diverse question-answer pairs.Source: §11 · p. 11
• GPT-4 combines concise ground-truth captions with detailed key-frame captions from LLaVA-v1.6 to generate dense video captions.The prompt instructs GPT-4 to eliminate conflicting information; Figure 8 illustrates the prompt.Source: §11 · p. 11, Figure 8 · p. 14
• GPT-3.5 generates VCG+ 112K instruction-tuning question-answer pairs using the ground-truth caption and dense video description.Figure 9 shows the prompt for generating detailed summary question-answer pairs.Source: §11 · p. 11, Figure 9 · p. 14
• VCGBench-Diverse prompts generate temporal, spatial, and reasoning questions, with similar prompts for consistency and summary questions.Figure 10 illustrates the temporal, spatial, and reasoning question-answer generation prompt.Source: §11 · p. 11, Figure 10 · p. 15
• Figures 6 and 7 compare VideoGPT+ with VideoChat2, GPT-4V, and Gemini-1.5-Pro-V across traffic, education, surveillance, sports, news, and automobile videos.Source: Figure 6 · p. 12, Figure 7 · p. 13