EVLM: An Efficient Vision-Language Model for Visual Understanding

1 Introduction

• Vision-language models handle tasks including object recognition, localization, OCR, and document analysis, while detail-focused methods improve OCR and document understanding.Some methods use multiple visual encoders; others raise image resolution or slice high-resolution images.Source: §1 · p. 1
• Long visual-token sequences raise computational overhead when fed directly into language models, while feature compression may lose visual information.This trade-off is especially relevant for video and multiple-image inputs.Source: §1 · p. 2
• EVLM uses Flamingo-like cross-attention, hierarchical ViT features, and MoE, and reports competitive public-benchmark results and strong image and video captioning.The model is pretrained on large-scale bilingual image-text pairs.Source: §1 · p. 2

2 Model Architecture

• EVLM uses a Flamingo-based architecture with a visual encoder, a large language model, and gated cross-attention layers.The architecture extracts hierarchical features to provide finer-grained visual signals.Source: §2 · p. 2
• Eight visual feature sequences sampled from the visual encoder’s last 40 layers are fed sequentially into different gated cross-attention layers.The visual encoder is EVA2-CLIP-E-Plus.Source: §2 · p. 3
• Learnable tokens replace the media token, and attention masks restrict each token set to its corresponding image and prior-image context.Zero vectors are padded onto the visual feature sequence for text not associated with visual features.Source: §2 · p. 3
• A gated cross-attention layer is inserted before every language-model transformer layer to condition the model on visual inputs.The language model is Qwen-14B-Chat 1.0.Source: §2 · p. 3
• The compute analysis compares concatenation-based full attention with cross-attention and estimates their FLOPs within an LLM transformer layer.It defines h_llm as the LLM hidden-state size and d_img as the visual-representation dimension.Source: §2 · p. 3
• SP was 0.24 with s_img=256 and s_txt=64 in multimodal pre-training, while SCP was 0.077 with s_img=1024 and s_txt=64 in continual pre-training.The paper presents these ratios as evidence of improved training efficiency.Source: §2 · p. 4

3 Training

• Training has three stages: multimodal pre-training, multi-task continual pre-training, and multimodal instruction fine-tuning.The stages progress from pre-training through continued capability acquisition to instruction tuning.Source: §3 · p. 4, §3.2 · p. 5, §3.3.1 · p. 6

3.1 Multi-modal Pre-training

• Pre-training uses image-text captions and web-type multimodal data, including 2.5 billion caption pairs and 50 million web-type examples, with 60% of the data in Chinese.Samples are packed to up to 64 images and 2,048 tokens; gated cross-attention is trained in the first 25% of training, then the latter half of the visual encoder is unfrozen for the remaining 75%.Source: §3.1 · pp. 4–5

3.2 Multi-task Continual Pre-training

• Continual pre-training targets higher-level visual question answering and combines VQA, NLP, OCR, detection, and replay data to prevent catastrophic forgetting.Training increases image resolution from 224 × 224 to 448 × 448 and yields EVLM-Base.Source: §3.2 · p. 5

3.3.1 Dense Baseline Model

• The dense baseline is instruction-finetuned on 2.3 million samples and then trains only cross-attention and the last quarter of ViT layers, producing EVLM-Chat.The instruction data includes user instructions, document and chart data, and math problems.Source: §3.3.1 · p. 6

3.3.2 Scaling via Mixture-of-Experts

• The MoE variant divides replicated FFN parameters into fine-grained experts and routes each token to selected experts, alongside a world expert used for every token.The configuration uses n=4, m=4, and k=4; it freezes the LLM and trains cross-attention and the last quarter of ViT layers to produce EVLM-MoE.Source: §3.3.2 · p. 6

4.1 Convergence of Multi-modal Pre-training Stage

• ImageNet-1K recognition accuracy improves during pre-training, and fine-grained accuracy rises steeply; Star accuracy remains relatively low despite extensive pre-training.The authors suggest the pre-training data may not cover enough multimodal knowledge for Star; coarse-grained alignment is possible with limited image-text data, unlike comprehensive fine-grained understanding.Source: §4.1 · p. 7

4.2 Comparison with State-of-the-Art VLMs

• EVLM-Chat and EVLM-MoE outperform competitors across most evaluated benchmarks, with reported strengths on general VQA, text-oriented VQA, MMBench variants, and POPE.The comparison covers 13 multimodal benchmarks.Source: Table 4 · p. 8, §4.2 · p. 9
• EVLM-Chat and EVLM-MoE achieve 86.4% and 86.8% accuracy on ScienceQA, respectively; on AI2Diagram, EVLM-MoE and EVLM-Chat achieve 75.5% and 76.0%, respectively.Source: §4.2 · p. 9

4.3 Image Caption

• Dense-caption descriptions are generated through multiple models, checked for authenticity, recombined, and refined before use in supervised fine-tuning.The authors report that fine-tuning with these descriptions reduces hallucinations in image descriptions.Source: §4.3 · p. 10

4.4.1 Attention Mask

• During video instruction fine-tuning, text tokens access all video visual features, while media tokens access only features from their corresponding frames.The attention mask is designed to avoid interference among information from different images in a sequence.Source: §4.4.1 · p. 10

4.4.2 Evaluation Benchmarks

• The video-dense-caption evaluation covers 3,596 videos across 39 categories, with GPT-4o used to score outputs.The evaluation set is assembled from publicly available data sources.Source: §4.4.2 · pp. 10–11

4.4.3 Caption Analysis

• EVLM generates dense video captions in Chinese and English that describe characters’ actions, environments, and action categories.Source: §4.4.3 · p. 11

• MLLM inputs use MLP-concatenated visual features (Liu et al., 2023a; Chen et al., 2024), q-formers (Zhu et al., 2023; Li et al., 2023a), or per-layer feature integration (Alayrac et al., 2022; Wang et al., 2023).Source: §5 · p. 11
• Vision encoders include CLIP, EvaCLIP, and SigLip; some work adds DINOv2 to mitigate potential CLIP-feature information loss, while other work uses lightweight CNNs for resolution and computation needs.The cited examples include CLIP (Radford et al., 2021), EvaCLIP (Sun et al., 2023), SigLip (Zhai et al., 2023), DINOv2 (Oquab et al., 2023), and CNNs (He et al., 2016).Source: §5 · p. 11
• MoE’s sparse activation can expand model or dataset scale at fixed compute; Upcycling trains MoE from dense models to reduce training costs, while fine-grained MoE targets expert specialization.The passage also describes MoE applications in LLMs and MLLMs.Source: §5 · p. 11

6 Conclusion and Future Work

• The model achieves competitive public-benchmark results, particularly in image and video dense captioning.It is also described as efficiently handling large-scale image-text pre-training through its refined visual-input approach.Source: §6 · p. 12
• Using more powerful, larger-scale language models is another proposed direction for improving performance.Source: §6 · p. 12
• Exploring video understanding with cross-attention under extremely long sequences remains a future direction.Source: §6 · p. 12

A.1 Caption data and Web-type data

• Caption-data cleaning included removing date-related text, converting traditional Chinese to simplified Chinese, and filtering low-relevance image-text pairs with CLIP.Source: §A.1 · p. 19
• Additional text filtering removed strings consisting only of numbers or symbols, long digit sequences, duplicate words, specified web terms, and excessively short text.Source: §A.1 · p. 19
• Web-type data processing removed damaged, solid-color, abnormal-aspect-ratio, and extremely low-resolution images, as well as records with excessive images or text over 2048 characters.Source: §A.1 · p. 19
• Web-type image-text pairs were also relevance-filtered to retain highly correlated images, using a process similar to MMC4 (Zhu et al., 2024).Source: §A.1 · p. 19

A.2 OCR

• The real-world OCR collection drew on Kuaishou videos, Wukong, Common Crawl 2021, street-view data, and image-based ebooks; processing extracted text and boxes, inpainted very small or low-confidence text, and filtered inadequate or redundant examples.Source: §A.2 · pp. 19–20
• Synthetic OCR data used SynthDog with text-free Kuaishou video frames, varied Chinese and English fonts, uncommon characters, artistic fonts, and dense LaTeX-generated text rendered from PDFs.Source: §A.2 · p. 20

B Hyperparameters

• Table 6 presents the training hyperparameter settings.Source: B Hyperparameters · p. 20, Table 6 · p. 20
• Multi-modal pre-training used AdamW, a cosine learning-rate schedule from 6e−4 to 3e−5, 2000 warm-up steps, weight decay 5e−2, and gradient clipping at 10.0.Training used 60 million samples at 224 × 224 pixels.Source: B Hyperparameters · p. 20
• Training first updated only Gated Cross Attention Layer parameters for 25% of pre-training, then unfroze the latter half of the Visual Encoder for the remaining 75%.Source: B Hyperparameters · p. 20
• Continual multi-task training increased visual-encoder input resolution from 224 × 224 to 448 × 448 to mitigate information loss from down-sampling.Its cosine learning rate ranged from 1e−4 to 5e−5, with 2000 warm-up steps.Source: B Hyperparameters · p. 21

C.1 Convergence of Multi-modal Pre-training Stage

• Figure 10 visualizes accuracy across seven fine-grained categories throughout the training process.Source: §C.1 · p. 21, Figure 10 · p. 21