PaperlayerView exact revision on arXiv

Source-linked AI summary

OmChat: A Recipe to Train Multimodal Language Models with Strong Long Context and Video Understanding

Tiancheng Zhao, Qianqian Zhang, Kyusong Lee, Peng Liu, Lu Zhang, Chunxin Fang, Jiajia Liao, Kelei Jiang, Yibo Ma, Ruochen Xu

arXiv:2407.04923v1cs.CVcs.CL

TL;DR

Long multimodal contexts, including videos and sequences of images, are challenging to manage and understand. OmChat combines unified visual processing, progressive multimodal training, and high-quality data selection. It supports contexts up to 512K tokens and reports strong performance on multi-image and video benchmarks, as well as competitive results on single-image benchmarks.

  • Problem

    Efficiently managing and understanding long multimodal contexts, including sequences of images and video frames, remains a key challenge.

    Source: §1 · p. 1

  • Method

    OmChat combines unified visual processing with active progressive multimodal pretraining and high-quality data selection.

    Source: §1 · p. 1, §2 · p. 3, §2.3 · p. 4

  • Results

    OmChat supports up to 512K tokens and outperforms other open-source models on multi-image and video benchmarks, while its 8B model achieves competitive single-image benchmark performance.

    Source: §1 · pp. 1–2, §6 · p. 14

  • Takeaways & Limitations

    The paper identifies higher image resolutions, progressive pretraining, and high-quality supervised fine-tuning datasets as factors associated with OmChat’s visual-understanding performance.

    Source: §1 · p. 2, §6 · p. 14

    Evaluation comparisons are scoped to benchmarks excluded from both pretraining and instruction tuning.

    Source: §3.2 · p. 8

Abstract

from arXiv · show

We introduce OmChat, a model designed to excel in handling long contexts and video understanding tasks. OmChat's new architecture standardizes how different visual inputs are processed, making it more efficient and adaptable. It uses a dynamic vision encoding process to effectively handle images of various resolutions, capturing fine details across a range of image qualities. OmChat utilizes an active progressive multimodal pretraining strategy, which gradually increases the model's capacity for long contexts and enhances its overall abilities. By selecting high-quality data during training, OmChat learns from the most relevant and informative data points. With support for a context length of up to 512K, OmChat demonstrates promising performance in tasks involving multiple images and videos, outperforming most open-source models in these benchmarks. Additionally, OmChat proposes a prompting strategy for unifying complex multimodal inputs including single image text, multi-image text and videos, and achieving competitive performance on single-image benchmarks. To further evaluate the model's capabilities, we proposed a benchmark dataset named Temporal Visual Needle in a Haystack. This dataset assesses OmChat's ability to comprehend temporal visual details within long videos. Our analysis highlights several key factors contributing to OmChat's success: support for any-aspect high image resolution, the active progressive pretraining strategy, and high-quality supervised fine-tuning datasets. This report provides a detailed overview of OmChat's capabilities and the strategies that enhance its performance in visual understanding.

1 INTRODUCTION

OmChat targets the challenge of handling long multimodal contexts, including sequences of images and video frames. It combines multimodal training and high-quality data selection, supports contexts up to 512K tokens, and reports strong results across multi-image, video, and single-image benchmarks.

1 INTRODUCTION

  • OmChat uses active progressive multimodal pretraining and selectively uses high-quality training data to build long-context and multimodal capabilities.The paper identifies higher image resolutions, progressive pretraining, and high-quality supervised fine-tuning data as factors in its visual-understanding performance.Source: §1 · pp. 1–2
  • OmChat supports up to 512K tokens and consistently outperforms other open-source models on multi-image and video benchmarks.The 8B model also achieves competitive single-image benchmark performance, often surpassing larger models.Source: §1 · pp. 1–2
  • Temporal Visual Needle in a Haystack evaluates whether OmChat can locate and interpret temporal visual details in long video sequences.The benchmark was proposed to assess temporal visual comprehension in videos.Source: §1 · p. 2
  • The paper presents OmChat’s architecture, training methodology, and evaluations across benchmarks, highlighting higher image resolutions, progressive pretraining, and high-quality data selection.Source: §1 · p. 2

2 METHOD

OmChat standardizes diverse visual inputs and combines dynamic vision encoding with staged training to support fine-grained perception and long contexts. Its training recipe progressively scales context and selects informative multimodal and instruction-tuning data.

2 METHOD

  • OmChat decomposes visual inputs into images for a shared processing path through the vision tower and language model, producing text outputs.The inputs include single or multiple images, image-text interleavings, and video frames; the architecture and training steps are shown in Figure 1.Source: §2 · pp. 2–3, Figure 1 · p. 2
  • Training first aligns a projector while freezing the vision tower and language model, then jointly optimizes all three components with a text-token cross-entropy objective.Source: §2 · p. 3

2.1 DYNAMIC VISION ENCODING

  • Dynamic vision encoding handles varying image resolutions while preserving fine details, and delimiters distinguish image patches, images, and video frames.The approach adopts AnyRes for variable-resolution image and video processing.Source: §2.1 · p. 3, §2 · p. 3

2.2 PROGRESSIVE TRAINING STRATEGY TO LONG CONTEXT

  • Progressive training extends context through 4K, 32K, 128K, and 512K stages, with RingAttention used for training beyond 128K.Shorter samples may be concatenated to fill target lengths; RoPE θ is scaled to 50M.Source: §2.2 · pp. 3–4

2.3 HIGH-QUALITY MULTIMODAL PRETRAIN DATA SELECTION

  • Selective Visual Language Modeling prioritizes image-relevant text and disregards text conflicting with image content by ranking tokens using excess loss.The reported average across six benchmarks rises from 32.7 to 38 after its use.Source: §2.3 · p. 4

2.4 HIGH-QUALITY INSTRUCTION TUNING DATA SELECTION

  • OmChat curates over 60 instruction-tuning datasets spanning text-only, single-image, multi-image, and video tasks.The collection includes tasks such as image comparison and video question answering to address multi-image scenarios and temporal relationships among frames.Source: §2.4 · p. 5
  • Continuous training evaluates instruction-tuning data portions by carrying checkpoints forward and adding a 10% linear-decay phase on each portion.Dataset proportions are manually set, and the procedure selects combinations without retraining earlier portions; Figure 2 illustrates the checkpoints and extra training steps.Source: §2.4 · p. 5, Figure 2 · p. 5

3 TRAINING DATA RECIPE

OmChat’s training data recipe combines varied pretraining sources with task-focused instruction-tuning data. The evaluation benchmarks are excluded from both training stages to prevent contamination.

3 TRAINING DATA RECIPE

  • The recipe separates broad pretraining data for foundational cross-modal understanding and language capability from smaller instruction-tuning data for downstream tasks.Source: §3 · p. 6

3.1 PRETRAINING DATA

  • Pretraining draws on image-text interleavings, OCR, video, captions, text-only sources, and visual question answering, alongside grounding and localization data.The data are drawn from public and proprietary sources, and the paper describes cleaning for sensitive information, duplicates, and damaged images.Source: §3.1 · pp. 6–7

3.2 INSTRUCTION TUNING DATA

  • Instruction tuning covers text-only, single-image, multi-image, and video tasks, including multi-image reasoning and video comprehension.The listed tasks also include logic and math, image comparison, knowledge QA, chart and figure understanding, OCR, and table understanding.Source: §3.2 · pp. 7–8
  • Evaluation benchmarks are excluded from pretraining and instruction tuning to avoid overlap between training and test data.Source: §3.2 · p. 8

4 EXPERIMENTS

OmChat is evaluated on single-image, multi-image, and video benchmarks, including long-context text and temporal visual needle tasks. Results report strong single-image performance and comparisons across multi-image and video benchmarks.

4.1 BENCHMARKS FOR EVALUATION

  • Evaluation spans single-image, math, OCR, multi-image, and video benchmarks, including MMBench, MathVista, OCRBench, Mantis-Eval, MileBench, and MVBench.MileBench includes long-context recall and realistic temporal and semantic multi-image tasks; MVBench covers 20 video-understanding tasks.Source: §4.1 · pp. 8–9

4.2 SINGLE IMAGE RESULTS

  • 78.8 on MMBench was OmChat's highest score, and the 8B model outperformed several larger models in single-image inference.The cited larger models include LLaVA-Next-Yi-34B, 360VL-70B, CogVLM-17B-Chat, XVERSE-V, IDEFICS2, and Yi-VL 34B.Source: §4.2 · p. 9

4.3.1 TEXT NEEDLE-IN-THE-HAYSTACK

  • Nearly perfect text-needle retrieval held from 4k to 256k tokens, with a minor decline at 512k attributed likely to shorter-context fine-tuning.The authors suggest that more long-context fine-tuning data should help improve performance at 512k.Source: Figure 3 · p. 10, §4.3.1 · p. 10

4.3.2 TEMPORAL VISUAL NEEDLE-IN-THE-HAYSTACK

  • TV Needle tests temporal visual comprehension by embedding emojis in video frames and asking models to locate visual needles with temporal information.The test videos concatenate 10–16 ActivityNet videos; three emojis are inserted into consecutive frames at varying depths, with inputs spanning 8k–256k tokens.Source: §4.3.2 · pp. 10–11
  • OmChat performed consistently across TV Needle input lengths and needle positions, surpassing Random and LLaVa-1.5; GPT-4o deteriorated at 222 frames and could not process 444 frames due to API constraints.The authors attribute OmChat's long-video capability to progressive training.Source: Figure 4 · p. 11, §4.3.2 · p. 11

4.3.3 MULTI-IMAGE AND VIDEO BENCHMARKS

  • OmChat scored more than 10 points higher on average than single-image-trained models across multi-image and video benchmarks, and outperformed Mantis and VILA.It also scored better than GPT-4V on MVBench, using 16 frames for OmChat's MVBench evaluation.Source: Table 2 · p. 11, §4.3.3 · p. 12

5 ABLATION & ANALYSIS

The ablations compare multimodal input formats and instruction-tuning data portions, then give qualitative examples of OmChat’s visual reasoning. F4 performs best across the tested benchmarks, while data selection balances multimodal and text-only performance.

5.1 MULTI-IMAGE AND VIDEO FORMAT ABLATION

  • F4 delivers the best performance across all benchmarks, using “image {i}” and the <im start>/<im end> delimiters; “image {i}” suits semantic tasks, while “frame {i}” scores highly on temporal video tasks.Table 3 compares formats F0–F4; the study also reports that F0 outperforms other single-image models on multi-image and video benchmarks.Source: Table 3 · p. 12, §5.1 · p. 13

5.2 DATA SELECTION ABLATION

  • The 20M-token data portion is selected as the best instruction-tuning combination: average scores rise from 10M to 20M, then fall from 40M to 80M as text-only performance declines while multimodal scores increase slightly.The authors suggest excessive multimodal training steps may explain the text-performance drop; Table 4 reports total averages across text and multimodal benchmarks.Source: §5.2 · p. 13, Table 4 · p. 13

5.3 QUALITATIVE EXAMPLES

  • Qualitative examples show OmChat predicting future video actions, making common-sense judgments, and comparing visual features across consecutive images.Figure 6 presents comparisons with open-source models; Figures 7 and 8 provide more task examples in the appendix.Source: §5.3 · pp. 13–14, Figure 6 · p. 14

6 CONCLUSION

OmChat combines progressive multimodal pretraining, high-quality supervised fine-tuning data, and support for high-resolution images. It supports contexts up to 512K tokens and performs strongly on multi-image and video tasks.

6 CONCLUSION

  • OmChat combines active progressive multimodal pretraining, high-quality supervised fine-tuning datasets, and any-aspect high image resolutions.The paper identifies these strategies as supporting performance across benchmarks.Source: §6 · p. 14
  • OmChat performs outstandingly on multi-image and video tasks while supporting context lengths up to 512K tokens.The conclusion highlights higher image resolutions, progressive multimodal pretraining, and high-quality training data as important to state-of-the-art performance.Source: §6 · p. 14
  • The authors describe OmChat as setting a new benchmark and identify efficient training techniques and broader multimodal applications as future work.Source: §6 · p. 14

A APPENDIX

The appendix includes further examples of OmChat handling varied tasks.

A APPENDIX

Related papers

Loading 2407.04923v1…