PaperlayerView exact revision on arXiv

Source-linked AI summary

Apollo: An Exploration of Video Understanding in Large Multimodal Models

Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, Serena Yeung-Levy, Xide Xia

arXiv:2412.10360v1cs.CVcs.AI

TL;DR

Video-LMM design remains poorly understood, while its computational demands and complex design space make systematic study challenging. This paper tests how design decisions transfer across model sizes, explores video-specific choices, and introduces Apollo, whose models outperform larger-model groups on video benchmarks.

  • Problem

    Video-LMMs remain underexplored, with high computational demands and a complex design space, leaving the factors driving video understanding poorly understood.

    Source: §1 · pp. 1–2

  • Method

    The paper studies transfer of design decisions across model sizes and explores video sampling, encoding, token resampling, integration strategies, and data composition.

    Source: §1 · p. 2

  • Results

    Apollo-3B outperforms most 7B models, and Apollo-7B outperforms all 7B models and many recent 30B models; Apollo-7B scores 70.9 (+5.4) on MLVU.

    Source: §1 · p. 2, §8 · p. 14

  • Takeaways & Limitations

    Design decisions made on smaller models can transfer to larger ones, reducing computational costs and enabling efficient experimentation.

    Source: §1 · p. 2

    The study excludes query-guided active frame selection and active token pooling, which require resampling at each conversational turn.

    Source: §4.1 · p. 7, §4.3 · p. 9

Abstract

from arXiv · show

Despite the rapid integration of video perception capabilities into Large Multimodal Models (LMMs), the underlying mechanisms driving their video understanding remain poorly understood. Consequently, many design decisions in this domain are made without proper justification or analysis. The high computational cost of training and evaluating such models, coupled with limited open research, hinders the development of video-LMMs. To address this, we present a comprehensive study that helps uncover what effectively drives video understanding in LMMs. We begin by critically examining the primary contributors to the high computational requirements associated with video-LMM research and discover Scaling Consistency, wherein design and training decisions made on smaller models and datasets (up to a critical size) effectively transfer to larger models. Leveraging these insights, we explored many video-specific aspects of video-LMMs, including video sampling, architectures, data composition, training schedules, and more. For example, we demonstrated that fps sampling during training is vastly preferable to uniform frame sampling and which vision encoders are the best for video representation. Guided by these findings, we introduce Apollo, a state-of-the-art family of LMMs that achieve superior performance across different model sizes. Our models can perceive hour-long videos efficiently, with Apollo-3B outperforming most existing $7$B models with an impressive 55.1 on LongVideoBench. Apollo-7B is state-of-the-art compared to 7B LMMs with a 70.9 on MLVU, and 63.3 on Video-MME.

1 Introduction

Video-LMM research faces high computational demands and an underexplored design space. The paper studies which design choices matter, identifies Scaling Consistency, and introduces Apollo, a family of video-LMMs with strong results across model sizes.

1 Introduction

  • The paper’s motivation is that video-LMM design choices remain poorly understood despite video’s richer temporal and spatial information than static images.The introduction highlights unresolved questions about sampling, vision encoders, and video-token resampling.Source: §1 · p. 1
  • The study explores video sampling, encoding, token resampling and integration, and data composition, finding that fps sampling outperforms uniform sampling and Perceiver Resampler outperforms average pooling.It also identifies robust vision-encoder combinations and finds that keeping about 10% text data during fine-tuning is critical for video understanding.Source: §1 · p. 2, Figure 1 · p. 2
  • The paper curates ApolloBench to address evaluation inefficiency and reports that many benchmark performance improvements are driven primarily by language modeling.The introduction describes ApolloBench as a subset of existing benchmarks that reduces evaluation time by 41×.Source: §1 · p. 2
  • Apollo-3B surpasses most existing 7B models, while Apollo-7B achieves 61.2 (+0.6) on Video-MME (w/o sub.), 70.9 (+5.4) on MLVU, and 66.3 (+2.4) on ApolloBench.The models are designed to comprehend hour-long videos.Source: §1 · p. 2
  • Scaling Consistency describes how design decisions effective for smaller LMMs and datasets transfer to larger ones, reducing computational costs and enabling efficient experimentation.The study examines the high computational demands and complex design space of video-LMMs.Source: §1 · pp. 1–2

2 How effective are existing video question-answering benchmarks?

The section examines whether video question-answering benchmarks measure video perception, how redundant their evaluations are, and how ApolloBench can provide a more efficient alternative.

2 How effective are existing video question-answering benchmarks?

  • Evaluating a 3B-parameter model across video question-answering benchmarks requires 184 A100 GPU hours, motivating analysis of benchmark quality and redundancy.The section evaluates ten open-source LMMs under video, center-frame image, and text-only settings.Source: §2 · p. 3, §2.1 · p. 3

2.1 Evaluating benchmark quality

  • Existing benchmarks often permit answers from text comprehension or a center frame alone; Video-MME (Short), MLVU, and TempCompass are the most discriminative among those examined.NExTQA and PerceptionTest can be solved using a single frame, while video-perception reliance decreases for longer videos.Source: §2.1 · pp. 3–4

2.2 Redundancy in existing benchmarks

  • Benchmark scores show substantial redundancy: duration groups correlate strongly within Video-MME and LongVideoBench, and TempCompass question formats have R2 > 0.8.The reported correlations are R2 = 0.94 between short and long Video-MME groups and R2 > 0.92 among LongVideoBench duration groups.Source: §2.2 · p. 4

2.3 Introducing ApolloBench

  • 41× faster evaluation is reported for ApolloBench, which retains high correlation with existing benchmarks while emphasizing video perception.It selects 400 discriminative questions across five temporal-perception categories and manually verifies them.Source: §2.3 · p. 4

3 Scaling Consistency: How small can you go during model design?

The paper tests whether smaller LMMs and datasets can inform design decisions for larger models. It finds a transfer threshold around 2–4B parameters and reports that about 500K samples suffice for reliable transfer from moderately sized models.

3 Scaling Consistency: How small can you go during model design?

  • The study trains 84 models from 21 design variations across four language-model sizes to measure how design choices transfer across scales.The variations include architecture, video sampling, training strategies, and data mixtures.Source: §3 · p. 4
  • R2 exceeds 0.9 for design decisions made on models of about 2–4B parameters, and the 4B-to-7B correlation is 0.938.The figure shows that correlation with 7B models rises with model size for 7B comparisons, unlike the trend for 0.5B models.Source: Figure 3 · p. 5, §3 · p. 5
  • Scaling Consistency offers a way to make design choices without extensive scaling studies, reducing computational burden and accelerating research.The paper contrasts this approach with traditional scaling laws, which require training models of various sizes for each design decision.Source: §3 · pp. 4–5
  • For 4B models, correlation with larger models plateaus at about 500K training samples, while 0.5B and 1.5B models show less consistent behavior across dataset sizes.The study varies dataset size from 75K to 1M samples.Source: §3 · p. 5

4 Exploring the video-LMM design space: what influences effective model design?

This section tests video-LMM design across sampling, representation, token resampling, and token integration, using Qwen2.5 3B and 750K samples; results correlate strongly with larger models and other model families.

4.2 Video representation

  • SigLIP-SO400M is the strongest single encoder, while InternVideo2 + SigLIP-SO400M is best overall; video encoders lead only on Temporal Perception.The combined encoders improve ApolloBench by approximately 7%.Source: Figure 5 · p. 7, §4.2 · p. 8

4.1 Video sampling

  • fps sampling is preferred over uniform sampling; across settings, 8–32 tokens per frame balance frame rate and tokens per second.The reported trade-off spans short- and long-video performance.Source: §4.1 · p. 7

4.4 Video token integration

  • 2–3% improvement across ApolloBench results from adding text or learnable tokens between video tokens; clip timestamps are best overall and require no new embeddings.The table caption identifies textual timestamps before each clip as the best-performing integration strategy.Source: §4.4 · p. 9, Table 2 · p. 10

4.3 Video token resampling

  • The Perceiver Resampler outperforms alternatives across metrics; channel-wise concatenation before resampling helps it integrate spatially aligned encoder features.The reported advantage was also observed when reducing tokens per frame.Source: Table 1 · p. 9, §4.3 · p. 9

5 How should video-LMMs be trained?

The section compares training schedules, video-encoder training choices, and data mixtures to identify effective video-LMM training protocols.

5.1 Training schedules

  • Three-stage training performs best, closely followed by two-stage training, with components progressively unfrozen across stages.When the LLM is frozen, other components are trained only on video; when it is tuned, training uses a text, image, multi-image, and video mixture.Source: §5.1 · p. 10, Table 3 · p. 11

5.2 Training video encoders

  • The experiments compare training encoders on data mixtures or video-only data, and whether aligning the connector first improves performance.When the LLM is frozen, training uses video data; when it is unfrozen, the mixture includes text, image, multi-image, and video.Source: §5.2 · p. 10
  • Training video encoders on image and video data while the LLM is unfrozen significantly hurts performance, whereas video-only encoder fine-tuning improves overall results, especially on reasoning and domain-specific tasks.Encoder training improves egocentric reasoning while most other metrics remain largely unaffected, likely due to better fine-grained vision-language alignment.Source: §5.2 · pp. 10–11

5.3 Data composition

  • Including 10–14% text data and a slightly video-heavy mix performs best; text proportions below 7% or above 14% harm performance.Among the tested 14% mixtures, the video-heavy mixture performs best.Source: §5.3 · p. 11, Figure 6 · p. 11

6 Apollo: a family of state-of-the-art large multimodal models

Apollo is a video-centric LMM family built from Qwen2.5 backbones and selected vision encoders, trained with a three-stage schedule. Across benchmarks, Apollo models compete with or surpass larger models.

6 Apollo: a family of state-of-the-art large multimodal models

  • Apollo uses 1.5B, 3B, and 7B Qwen2.5 backbones, SigLIP-SO400M and InternVideo2 encoders, and a Perceiver Resampler to produce 32 tokens per frame.Encoder features are interpolated, concatenated along the channel dimension, and resampled using a three-stage training schedule.Source: §6 · p. 12
  • Apollo’s training corpus combines publicly available and licensed text, image-text, multi-image, and video datasets, plus generated multi-turn video conversations.Licensing constraints led the authors to omit some commonly used datasets, including non-permissive sources.Source: §6 · p. 12
  • Apollo-3B scores 68.7 on MLVU, surpassing Oryx-7B’s 67.5, while Apollo-7B scores 70.9, narrowly above Oryx-34B’s 70.8.The table compares Apollo with proprietary and open-source models across video benchmarks and ApolloBench.Source: §6 · p. 12, Table 4 · p. 13

7 Background

Prior video-LMM research has expanded architectures, training data, and benchmarks, but the effects of many design choices remain insufficiently analyzed. This section situates the paper’s systematic study against earlier video- and image-LMM work.

7 Background

  • Earlier video-LMMs used sparsely sampled frames, MLP connectors, or training-free methods, while later systems added token resampling and only a few used video-specific encoders.Most approaches cited use image-based encoders rather than video-specific encoders.Source: §7 · p. 13
  • Video-LMM training and benchmarks have expanded from small instruction datasets and short-video tasks toward larger, higher-quality data and long-video evaluation, yet design decisions remain underanalyzed.The cited work includes multi-image datasets as one way to further enhance model capabilities.Source: §7 · p. 14
  • Image-LMM studies have examined encoders, training strategies, and data mixtures, but video-LMMs add distinct challenges in frame sampling, token resampling, encoder selection, and efficient evaluation.The paper frames its study as a systematic investigation of these video-specific design choices.Source: §7 · p. 14

8 Conclusion

The authors conclude that systematic design and training choices can yield strong video understanding without relying on larger models. They present Apollo and the study’s findings as resources for future video-LMM research.

8 Conclusion

  • The study examines architecture, training schedules, data mixtures, and evaluation, and presents Scaling Consistency and ApolloBench as ways to support smaller-model research and faster evaluation.The authors hope their insights into video sampling, encoder selection, token resampling, and token integration will further democratize video-LMM research.Source: §8 · p. 14
  • Apollo-3B outperforms most advanced 7B models, while Apollo-7B outperforms all 7B models and many recent 30B models.The conclusion attributes these results to careful design and training strategies rather than larger model sizes.Source: §8 · p. 14

Appendix

  • The appendix provides benchmark and ApolloBench analyses, implementation details, Scaling Consistency and sampling analyses, and raw experimental results.It includes raw data for the scaling, architecture, token-integration, training-schedule, and data-mixture studies.Source: Appendix · p. 20

A Future work

The authors identify several unresolved design questions for Apollo, including how image and video processing should be separated and how encoders should be trained. They also call for broader validation of Scaling Consistency and conversational evaluation.

A Future work

  • Apollo currently uses a unified visual pipeline; testing separate image and video encoders could reveal performance benefits and improve modality handling.The current architecture uses the video encoder for images as well, duplicating each image N times.Source: A Future work · p. 20, Figure 8 · p. 21
  • Training image and video encoders during SFT in a split architecture could help identify their individual contributions and optimal training strategies.The authors also propose mixed image-video training of both encoders in a unified architecture to investigate performance drops.Source: A Future work · p. 20
  • Scaling Consistency needs broader testing to establish whether it remains reliable for larger models.Source: A Future work · p. 21
  • The study leaves memory banks and frame-retrieval methods unexamined, including whether they generalize well to multi-turn conversations.Source: A Future work · p. 21
  • Existing benchmarks mainly use academic multiple-choice questions and inadequately assess conversational abilities, motivating a dedicated conversational benchmark.Source: A Future work · p. 21

B.1 Correlations within existing benchmarks

The section examines whether existing video benchmarks measure video perception consistently across durations and question formats, then describes ApolloBench’s curation. It reports that ApolloBench is faster to evaluate while retaining high correlation with existing benchmarks.

B.1 Correlations within existing benchmarks

  • R2 > 0.92 across LongVideoBench duration groups and > 0.83 across Video-MME groups, but longer-video performance relies increasingly on text rather than vision.On Video-MME, text and image performance remain mostly unchanged as video-modality performance decreases with duration.Source: §B.1 · pp. 21–22
  • R2 > 0.81 between question formats on TempCompass supports multiple-choice as a video-perception benchmark format, though it does not assess conversational ability.Source: §B.1 · p. 22, Figure 12 · p. 24

B.2 Raw evaluations

  • The raw evaluations compare open-source LMMs across six video benchmarks using video, image, and text inputs.The reported evaluations use lmms-eval; full model results are in Tables 5 and 6.Source: Table 5 · p. 21, Table 6 · p. 22, §B.2 · p. 23

B.3 ApolloBench curation

  • ApolloBench’s manually verified final dataset contains the 400 questions with highest entropy; it evaluates 41× faster than existing benchmarks while maintaining high correlation with their results.Questions answerable correctly by more than 50% of models using either modality were filtered out; the remaining questions were categorized into five temporal-perception categories.Source: §B.3 · pp. 23–24

C Apollo implementation details

C.1 Architecture

Apollo uses a shared image-video processing pipeline and adopts a unified architecture after finding it slightly better or on par with a split alternative. The section also describes its training data and training setup.

C.1 Architecture

  • Apollo encodes N-frame clips with InternVideo2 and SigLIP-SO400M, interpolates and concatenates their features, then projects and resamples them into T tokens per clip.For images, the shared pipeline duplicates the image N times; video clips are sampled at a constant fps and distributed across the video.Source: §C.1 · pp. 24–25

C.2 Unified vs. Split Architectures

  • The unified architecture performs slightly better or on par with the split architecture across key benchmarks, so Apollo adopts it as the default.The comparison is reported for ApolloBench across OCR, Spatial, Egocentric, Perception, Reasoning, and overall performance.Source: §C.2 · p. 25, Table 8 · p. 26

C.3 Data

  • Apollo’s training data spans text, image-text, multi-image, and video modalities, but licensing restrictions excluded some commonly used datasets.Training proceeds through alignment, video-only vision pretraining, and supervised fine-tuning on a mixed-modality dataset.Source: §C.3 · p. 26

C.4 Training

  • Training used 128 NVIDIA A100 GPUs, AdamW, gradient clipping at 1, a 0.03 warm-up ratio, and a cosine learning-rate schedule.The objective was autoregressive text-generation cross-entropy; a higher connector learning rate performed best.Source: §C.4 · pp. 26–27

D Scaling Consistency: efficient model design with smaller models

The paper proposes Scaling Consistency: design choices tested on moderately sized LMMs can correlate with choices on larger models. It also finds that video-sampling needs vary by perception task.

D Scaling Consistency: efficient model design with smaller models

E Effect of video sampling on the different dimensions of video perception

  • OCR and spatial tasks lose performance steeply as tps falls, while egocentric understanding and reasoning are more sensitive to fps; perception often favors lower fps.The authors conclude that sampling strategies should be tailored to the requirements of each perception task.Source: E Effect of video sampling on the… · p. 27

F Raw results

This appendix provides the study’s raw evaluation data for further analysis, with tables covering video sampling, vision encoders, training schedules, data mixtures, and scaling consistency.

F Raw results

  • The study provides raw evaluations of all models and raw data for further analysis; Sections 3, 4.1, 4.2, 4.4, 5.1, and 5.3 map to Tables 14 and 15, 10 and 11, 9, 2, 12, and 13, respectively.Source: F Raw results · p. 28
  • Table 9 compares single-image, single-video, and dual vision encoders on six ApolloBench metrics at 32 TPS.The table groups and orders encoder configurations by type.Source: Table 9 · p. 29
  • Tables 10 and 11 report sampling results across six ApolloBench metrics, covering fps, tps, and tpf as well as uniform-sampling train-test conditions.Table 11 separates models trained and tested with uniform sampling from models trained uniformly and tested at 2 fps.Source: Table 10 · p. 29, Table 11 · p. 30
  • Table 12 reports schedule results across one-, two-, and three-stage training, varying learning rates and data mixtures across six ApolloBench metrics.The caption notes benefits from multi-stage training and refers readers to Table 7 and Section C.3 for details.Source: Table 12 · p. 30
  • Table 13 compares video-LMM performance across Text, Image, Multi-Image, and Video data proportions on six ApolloBench metrics.The caption highlights the role of balanced data mixtures and refers readers to Section 5.3.Source: Table 13 · p. 31
  • Tables 14 and 15 report 42 Scaling Consistency model configurations spanning language-model size, vision towers, video settings, and data mixtures, with an Average performance score.The configurations support investigation of smaller models as proxies for larger models in design decisions.Source: Table 14 · p. 32, Table 15 · p. 33
Loading 2412.10360v1…