PaperlayerView exact revision on arXiv

Source-linked AI summary

Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders

Boqiang Zhang, Lei Ke, Ruihan Yang, Qi Gao, Tianyuan Qu, Rossell Chen, Dong Yu, Leoweiliang

arXiv:2603.06569v2cs.CV

TL;DR

Compact VLM deployment is constrained by large models and heavy training, while conventional contrastive vision encoders may inadequately support fine-grained multimodal understanding. Penguin-VL addresses this with a text-only-LLM-initialized vision encoder and a unified training recipe, achieving strong results across image and video benchmarks at compact scale.

  • Problem

    VLM progress commonly relies on large models and contrastively pretrained vision encoders, leaving efficient deployment and the suitability of this encoder paradigm insufficiently addressed.

  • Method

    Penguin-VL initializes its vision encoder from a text-only LLM and combines mixed-supervision pretraining, multimodal training, instruction tuning, and temporal redundancy-aware video compression.

  • Results

    Penguin-VL achieves strong performance across standard image and video benchmarks at compact scale, with leading results on most chart and document understanding benchmarks at 2B scale.

  • Takeaways & Limitations

    The results support LLM-initialized visual representation learning as a compact alternative to conventional contrastive-pretrained encoders for multimodal systems.

Abstract

from arXiv · show

Vision Language Model (VLM) development has largely relied on scaling model size, which hinders deployment on compute-constrained mobile and edge devices such as smartphones and robots. In this work, we explore the performance limits of compact (e.g., 2B and 8B) VLMs. We challenge the prevailing practice that state-of-the-art VLMs must rely on vision encoders initialized via massive contrastive pretraining (e.g., CLIP/SigLIP). We identify an objective mismatch: contrastive learning, optimized for discrimination, enforces coarse and category-level invariances that suppress fine-grained visual cues needed for dense captioning and complex VLM reasoning. To address this issue, we present Penguin-VL, whose vision encoder is initialized from a text-only LLM. Our experiments reveal that Penguin-Encoder serves as a superior alternative to traditional contrastive pretraining, unlocking a higher degree of visual fidelity and data efficiency for multimodal understanding. Across various image and video benchmarks, Penguin-VL achieves performance comparable to leading VLMs (e.g., Qwen3-VL) in mathematical reasoning and surpasses them in tasks such as document understanding, visual knowledge, and multi-perspective video understanding. Notably, these gains are achieved with a lightweight architecture, demonstrating that improved visual representation rather than model scaling is the primary driver of performance. Our ablations show that Penguin-Encoder consistently outperforms contrastive-pretrained encoders, preserving fine-grained spatial and temporal cues that are critical for dense perception and complex reasoning. This makes it a strong drop-in alternative for compute-efficient VLMs and enables high performance in resource-constrained settings. Code: https://github.com/tencent-ailab/Penguin-VL

1 Introduction

Penguin-VL targets compact, efficient multimodal systems by treating visual representation learning as a central bottleneck rather than relying primarily on larger models and heavier training. It introduces an LLM-initialized vision encoder and an integrated training recipe for fine-grained image and video understanding.

  • Leading VLMs often use massive parameter counts and heavy training pipelines, creating a gap between research prototypes and practical deployment.
  • Penguin-VL centers efficiency gains on the visual backbone, combining a new LLM-based encoder, progressive multimodal training, and curated data.
  • Contrastive pretraining is challenged as a default because its discriminative paradigm may provide insufficient fine-grained multimodal alignment for VLM encoders.
  • Penguin-Encoder reuses text-only LLM backbone weights to provide tighter modality alignment and richer initial linguistic knowledge.
  • The training recipe combines mixed supervision, a low-to-high-resolution curriculum, priority-aware video token compression, and two-stage instruction tuning.
  • Penguin-VL reports strong performance across image and video benchmarks while maintaining compact, computationally efficient model sizes.

2 Methodology

Penguin-VL combines an LLM-initialized vision encoder with a lightweight projector and a content-adaptive video compression strategy. Its encoder training uses alignment and reconstruction objectives to preserve visual detail while connecting visual features to language representations.

  • Architecture: Penguin-VL uses a three-module design comprising Penguin-Encoder, an MLP-based vision–language projector, and an LLM, with 2B and 8B VLM variants.Penguin-Encoder keeps a fixed parameter size independent of the VLM backbone.
  • LLM-based vision encoder: Penguin-Encoder is initialized from Qwen3-0.6B and adapted with bidirectional attention and 2D-RoPE for visual representation learning.The encoder processes variable-resolution inputs at their original spatial resolution when the token budget permits.
  • LLM-based vision encoder: The text-to-vision initialization provides architectural expressivity, native alignment with the decoder, and inherited semantic priors.These advantages are presented as practical benefits of reusing an LLM architecture and its pretrained knowledge.
  • Encoder training: Encoder training jointly uses language-model cross-entropy and reconstruction objectives, with amplitude, directional, and relational losses supervising visual features and inter-patch relationships.The reconstruction stage is intended to preserve fine-grained visual details and the overall visual-feature distribution.
  • Video encoding and projector: The model converts visual inputs into tokens, applies a fixed-rate or uniform frame-sampling policy when videos exceed max_frames, and uses an MLP projector to match the LLM hidden size.The projector avoids spatial token compression and directly transforms visual feature dimensions for language-model integration.
  • Video encoding and compression: TRA allocates more visual tokens to informative keyframes while limiting intermediate-frame tokens under a global budget.Its cascade preserves native resolution when possible, synchronously downscales frames when necessary, and clamps intermediate frames at Tmin before further compressing keyframes.

3.1 Data Format

Penguin-VL represents images and videos as unified token sequences with modality-specific separators and timestamp tags. Image sequences use newline-separated image blocks, while video sequences use timestamped frames and comma delimiters.

  • Sequence construction: Images and videos are converted into a single LLM token sequence using modality-specific content blocks, explicit separators, and absolute timestamps.Image blocks are denoted by I, video frames by V, and text tokens by X.
  • Separators: Newline separators distinguish distinct inputs, while commas delimit items within a continuous stream such as video frames.The format uses “\n” between separate images or final text and “,” between consecutive frames.
  • Timestamps: Timestamp tags use the format “Time: xxs” to represent each video frame’s absolute time.Each frame block is immediately preceded by its timestamp.
  • Image sequence: A sequence of N images is serialized as I1 \n I2 \n · · · \n IN \n X when followed by text instructions.The same newline convention separates the image blocks and the text instruction.
  • Video sequence: A video is serialized as timestamped frame blocks separated by commas and followed by a newline before the text instruction.This format preserves the temporal ordering of frames and their absolute timestamps.

3.2 Stage 1: Penguin-Encoder Training

Penguin-Encoder is initialized from a text-only LLM and trained through a coarse-to-fine visual-learning strategy. Low-resolution training bootstraps broad perception, while high-resolution fine-tuning emphasizes detailed spatial and semantic alignment.

  • Encoder initialization: Penguin-Encoder inherits its parameterization and architectural priors from a text-only LLM, using approximately 400M parameters for visual modeling.A randomly initialized two-layer GELU MLP serves as the vision projector, while the language decoder remains frozen during encoder training.
  • Low-resolution pre-training: The first training stage uses approximately 100M samples with resolution capped at 2048 visual tokens and primarily noisy original captions.Because captions are sparse or imprecise for charts and diagrams, the stage also augments supervision with unlabeled structured visual data.
  • High-resolution fine-tuning: High-resolution fine-tuning increases the input budget to 10240 visual tokens and uses filtered, high-quality re-captioned data.Removing the reconstruction branch focuses this stage on fine-grained alignment, detailed spatial structures, and subtle semantic correspondences.
  • Low-resolution pre-training: Low-resolution initialization totals roughly 223M samples, including approximately 220M image–text pairs and 2.8M unlabeled chart samples.The image–text pairs are sampled from COYO-700M and DataComp-1B, while chart data comes from four chart and diagram datasets.
  • High-resolution fine-tuning: The high-resolution stage samples approximately 45M images from diverse image and document sources and prioritizes high-resolution inputs through longer token sequences.The data are re-annotated to provide richer semantic supervision.

3.3 Stage 2: Pre-training

Stage 2 pre-training jointly updates the LLM, vision encoder, and projector using a diverse multimodal data mixture. The mixture combines broad caption supervision with document, fine-grained, scientific, OCR, code, mathematics, text, grounding, and region-caption data.

  • Pre-training: Stage 2 pre-training trains all parameters, including the LLM, vision encoder, and vision projector, to provide diverse multimodal knowledge.The pre-training data mixture contains approximately 121M samples.
  • Data mixture: General caption data comprise 64% of the mixture and broaden visual coverage while strengthening image–text alignment.Additional categories include document, fine-grained, scientific, OCR, code, math, and pure text data.
  • Data mixture: Document data support OCR and fine-grained visual recognition, while their typically high resolution also supports high-resolution visual processing.Document data are described as the largest category after general caption data.
  • Specialized supervision: Science, OCR, code, and pure-text data extend multidisciplinary knowledge, visual-text reasoning, programming–multimodal connections, and language retention.The paper states that text-only data mitigate catastrophic forgetting in the language component during multimodal training.
  • Region-level supervision: Fine-grained supervision includes grounding and region-caption annotations that connect spatial localization with semantic understanding.The grounding corpus contains 7.7 million samples, and region-caption supervision adds 1.5 million region-caption QA pairs.
  • Region-level supervision: Grounding coordinates are normalized to an integer [0, 1000] space, with the image corners represented by (0, 0) and (1000, 1000).Integer coordinates are used because they align with LLM tokenization and are empirically more stable for autoregressive prediction than decimal regression.

3.4 Stage 3: Supervised Fine-Tuning

Penguin-VL’s supervised fine-tuning uses diverse, balanced image and video instruction data to cover broad multimodal capabilities, from perception to complex reasoning.

  • The SFT stage aligns the model’s multimodal capabilities with user intent through image and video instruction tuning.
  • Image SFT Mixture: Image SFT data are cleaned and balanced across domains to support broad usage and reduce domain bias.The mixture covers general comprehension, documents, OCR, grounding, counting, mathematics, multi-image reasoning, and science.
  • Image SFT Mixture: Document, chart, table, OCR, and text-QA supervision targets layout understanding, structured interpretation, text recognition, and language-centric reasoning.Document, Chart & Table data comprise 20.9%, while OCR and Text QA comprise 16.6%.
  • Image SFT Mixture: Grounding, counting, mathematics, multi-image, and science data strengthen fine-grained correspondence, quantitative reasoning, and relational understanding.Grounding & Counting account for 10.1%, Mathematics for 8.9%, and Multi-image and Science are included in Others.
  • Video SFT Mixture: Video SFT balances general comprehension with action reasoning, temporal grounding, and egocentric understanding across four data domains.The proportions are 77.6% general video understanding, 12.7% action recognition and reasoning, 6.9% temporal grounding and reasoning, and 2.8% ego video understanding.

4 Experiment

The experiment section specifies Penguin-VL’s training, preprocessing, inference, and evaluation setup across compact 2B and 8B model variants.

  • Training uses cosine learning-rate decay with a 3% warm-up ratio and a maximum sequence length of 16,384 tokens.Up to 10,240 tokens are allocated to visual inputs.
  • Penguin-VL uses Qwen3-based 2B and 8B backbones with a Qwen3-0.6B vision encoder and a two-layer GELU MLP projector.The vision encoder is initialized from Qwen3-0.6B, while the language backbone matches the model scale.
  • Video processing applies spatial downsampling, frame sampling, and token compression to improve computational efficiency for extended inputs.Training preprocessing caps videos at 180 frames and applies Priority-Aware Visual Token Encoding and Compression.
  • Baselines: Evaluation compares Penguin-VL with compact multimodal baselines, including Gemma3n-E2B-it, SmolVLM2, InternVL3.5-2B, and Qwen3-VL-2B.The 8B configuration evaluates corresponding 8B variants when available.
  • Image and video inference use near-greedy deterministic decoding, while video evaluation additionally varies frame budgets and TRA sampling strategies.The reported Table 1 results use the best-performing configuration among the evaluated settings.

4.3 Image Benchmarks

Penguin-VL is evaluated on document, chart, scene-text, mathematical, logical, multi-image, general-knowledge, and scientific reasoning benchmarks, showing especially strong image understanding at both compact scales.

  • Penguin-VL’s image evaluation spans text and data interpretation, mathematical and logical reasoning, and cross-image synthesis grounded in general knowledge.
  • Document, Chart, and Scene Text Understanding: Document- and text-centric benchmarks test OCR, layout understanding, chart interpretation, numerical reasoning, and scene-text comprehension.The suite includes DocVQA, ChartQA, InfoVQA, and OCRBench.
  • Mathematical, logical, multi-image, general-knowledge, and scientific benchmarks assess visual deduction, abstract reasoning, cross-image understanding, and domain knowledge.The evaluation includes MathVista, MathVerse, LogicVista, MMMU-Pro, and related cross-image and scientific tasks.
  • 2B Model: The 2B model leads most chart and document benchmarks, achieves 41.29 on LogicVista, and reaches top rankings on three of five general-knowledge tasks.It trails Qwen3-VL and InternVL3.5 on OCRBench but surpasses earlier baselines by 81–110 points.
  • 8B Model: The 8B model achieves 96.2 on DocVQA, 90.5 on ChartQA, and 77.4 on MathVista, frequently surpassing Qwen3-VL in document and graphical interpretation.It is slightly behind Qwen3-VL on OCRBench and CharXiv while remaining ahead of other cited contemporaries.

4.4 Video Benchmarks

Across video benchmarks, Penguin-VL demonstrates strong short-form, long-form, temporal, and spatial-temporal reasoning, with particularly strong results at 2B and 8B scales.

  • Video evaluation covers general understanding, long-form comprehension, temporal reasoning, and temporal sentence grounding.Benchmarks include MVBench, VideoMME, EgoSchema, PerceptionTest, ActivityNetQA, MMVU, LongVideoBench, NextQA, and Charades-STA.
  • 2B Model: The 2B model leads EgoSchema at 57.6, ActivityNetQA at 61.5, Perception Test at 70.4, NextQA at 79.9, and Charades-STA at 56.2.It also ties for first on MMVU at 42.7 and surpasses Qwen3-VL on LongVideoBench by 7.4 points.
  • 8B Model: The 8B model leads most evaluated video metrics, including LongVideoBench at 67.0, NextQA at 85.4, ActivityNetQA at 65.2, and Perception Test at 78.0.It also reaches 61.4 on Charades-STA, outperforming Qwen3-VL by 5.4 points.
  • Ablation Study: Encoder ablations evaluate LLM initialization, relation loss, and reconstruction loss through a lightweight three-stage training pipeline.
  • Ablation Study: LLM initialization raises the average score from 31.3 with random initialization to 34.6, a +3.3 absolute improvement.The reported explanation is that pretrained LLM weights provide a well-conditioned starting point for modality alignment.
  • Ablation Study: Relation loss raises the average score from 33.3 with basic reconstruction objectives to 34.6.
  • Encoder Comparison: Penguin-encoder reaches an average score of 49.3 using approximately 240M pre-training samples and outperforms the compared vision encoders across all benchmarks.The matched-data experiment also reports Penguin-encoder as clearly superior to SigLIP2 under the same data and training recipe.
  • Encoder Comparison: As data scale and task complexity increase, Penguin-encoder shows progressively larger gains over contrastive encoders.

4.6 Image and Video Case Study

Penguin demonstrates broad image and video capabilities, spanning visual problem solving, document OCR, chart analysis, creative writing, and temporal video understanding. These cases emphasize fine-grained perception alongside formal reasoning and generation.

  • Logical reasoning and code writing: Penguin converts a visually presented hard programming problem into a dynamic-programming formulation and syntactically correct Python code.Its output handles wildcard-pattern edge cases through explicit state-transition logic.
  • OCR and dense document understanding: Penguin extracts text from degraded historical documents while preserving precise reading order and document structure.The case combines layout-aware OCR with downstream semantic-parsing prerequisites.
  • Chart image understanding: Penguin interprets multivariate line charts by locating fine-grained extrema and comparing volatility across distributions over a 50-year span.The example includes identifying global minima within a specified temporal window.
  • Image creative writing: Penguin transforms concrete visual details and historical atmosphere from a painting into coherent, contextually aligned creative writing.The case extends beyond object detection to implicit semantic and artistic interpretation.
  • Video understanding and temporal grounding: Penguin processes extended video sequences into global narratives while temporally grounding semantic queries to landmarks and key events.The evaluation covers both long-context comprehension and fine-grained temporal localization.

5 Related Work

Related work has largely advanced VLMs through unified high-resolution systems, contrastively pretrained vision encoders, resolution-aware visual processing, and temporal video feature compression. These approaches address multimodal capability, visual information preservation, and video token-length constraints.

  • Vision-centric multimodal LLMs: Recent VLMs increasingly pursue unified, high-resolution, and agentic multimodal systems with expanded context and multimodal processing.Examples include GPT-4o’s text, vision, and audio processing and Gemini’s long-video context windows.
  • Encoder design: Most earlier vision encoders use CLIP- or SigLIP-style contrastive image–text pretraining, either through pretrained models or custom ViT backbones.The shared paradigm trains vision transformers on paired image–text data with contrastive objectives.
  • Visual feature handling: Visual processing research uses tiling, patch-level methods, spatial encodings, and RoPE to preserve information across varying image and video resolutions.These methods respond to information loss caused by naïve resizing.
  • Video understanding: Video VLMs address dense-frame token growth and temporal redundancy through fixed frame budgets, pooling or MLP compression, and learnable temporal fusion.Temporal convolution and related modules aggregate information across frames.

6 Conclusion and Future Work

The paper presents Penguin-VL as a compact vision-centric model built around an LLM-initialized vision encoder and a three-stage training pipeline. It concludes that this approach supports parameter-efficient image and video understanding while identifying real-time inference, reinforcement-learning post-training, and agentic use as future directions.

  • Conclusion: Penguin-VL combines a text-only-LLM-initialized Penguin-Encoder with encoder training, VLM pretraining, SFT, and open-source data curation.The framework targets image and video understanding with a vision-centric design.
  • Conclusion: The paper reports parameter-efficient performance at the 2B and 8B scales across capabilities from document OCR to long-context temporal reasoning.The conclusion characterizes these evaluations as establishing a new standard for parameter-efficient VLMs.
  • Future work: Future work includes real-time inference optimization for low-latency operation under strict time and resource constraints.The proposed applications include embodied agents, GUI automation, and assistive systems.
  • Future work: Future work will explore reinforcement-learning post-training beyond SFT to optimize long-horizon objectives, interaction dynamics, and implicit user preferences.The motivation is that static annotations do not fully capture these objectives.
  • Future work: The paper identifies agentic visual-language systems as a direction for grounding perception and reasoning in GUI actions under partial observability.This requires perceiving interface elements, tracking state changes, and planning long-horizon actions.
Loading 2603.06569v2…