Source-linked AI summary
ENCORE: Entropy-Guided Cropping and Attention Regularization for Robust Vision--Language Understanding
Yuanhao Sun, Huawei Ji, Jiaxin Ding, Luoyi Fu, Xinbing Wang
TL;DR
Lightweight VLMs can fragment prompt-relevant regions when images are partitioned into fixed-resolution patches, and existing methods do not dynamically preserve their integrity. ENCORE uses early-layer image–text entropy for entropy-guided cropping and entropy-regularized training, achieving state-of-the-art performance among recent 2B-parameter VLMs with 0.14% trainable parameters.
Problem
Fixed-resolution image patching can fragment prompt-relevant regions, while existing methods focus on the visual modality and do not dynamically preserve those regions.
Method
ENCORE combines ECS, which selects the minimal-entropy crop ratio, with ERT, which uses entropy regularization to sharpen attention toward key visual tokens.
Results
ENCORE achieves state-of-the-art performance across ten VQA benchmarks among recent 2B-parameter VLMs, with 0.14% of parameters trainable.
Takeaways & Limitations
Image–text entropy provides a quantitative basis for preserving prompt-relevant visual regions and mitigating semantic fragmentation in VLMs.
Abstract
from arXiv · showhide
Vision-Language Models (VLMs) perform well on diverse vision-language tasks, but transformer-based visual encoders split images into fixed-resolution sub-images, compromising object integrity in lightweight VLMs. Existing methods only focus on the visual modality and fail to dynamically preserve the integrity of prompt-relevant regions, limiting performance. In this work, we observe that the early-layer image-text entropy of cross-modal attention strongly correlates with answer grounding quality and task accuracy. Building on this finding, we propose \textbf{ENCORE}, an entropy-guided framework with two components: At inference, an \textbf{Entropy-based Cropping Strategy} (ECS) evaluates a small set of candidate crops and selects the one with minimal entropy, preserving contiguous regions relevant to the prompt. At training, \textbf{Entropy Regularization Training} (ERT) augments next-token prediction with an entropy term that sharpens attention on key visual tokens while down-weighting irrelevant ones. Experiments on ten VQA benchmarks show that ENCORE, fine-tuning only 0.14\% of parameters, achieves an average 1.43\% accuracy gain and state-of-the-art performance among recent 2B-parameter VLMs. Our code is released in https://github.com/baokou-fw2/ENCORE.
1. INTRODUCTION
Lightweight VLMs can fragment prompt-relevant regions when high-resolution images are partitioned into patches, while existing visual-only remedies do not dynamically preserve those regions. ENCORE uses image–text entropy to guide cropping and training, achieving strong performance with parameter-efficient fine-tuning.
- High-resolution patch partitioning can split prompt-relevant objects, such as the word ‘Breakdown’, impairing holistic visual understanding.
- Existing cropping, resizing, thumbnail, and multi-scale methods remain limited because they focus on the visual modality rather than prompt-relevant region integrity.
- ENCORE combines entropy-guided cropping with entropy-regularized training to preserve prompt-relevant regions and improve key visual token extraction.
- 0.14% of model parameters are trainable via LoRA fine-tuning, while ENCORE achieves state-of-the-art performance across ten VQA benchmarks.
- Lower early-layer cross-modal entropy correlates with more accurate visual understanding and stronger answer grounding.
2. METHOD
ENCORE uses image–text entropy to quantify the integrity of prompt-relevant visual regions and guide both inference-time cropping and training-time attention sharpening. Its two components address semantic fragmentation caused by high-resolution image partitioning and mismatched visual–text pairs.
- Image–Text Entropy: Image–text entropy measures prompt-relevant visual-region fragmentation and provides a quantitative basis for analyzing and mitigating semantic fragmentation.A smaller HS corresponds to more contiguous preservation of prompt-relevant regions and correlates with answer-grounding coverage.
- Entropy-Based Cropping Strategy: ECS computes HS from early VLM layers, selects the candidate cropping ratio with minimal entropy, and combines the resulting crops with the originals.The strategy uses cross-modal similarities to derive prompt relevance and preserve contiguous prompt-relevant semantics.
- Motivation: High-resolution patching can split contiguous object regions, while next-token prediction does not explicitly suppress mismatched visual–text pairs.This makes it difficult to distinguish key visual tokens from irrelevant ones in fragmented inputs.
- Entropy Regularization Training: ERT adds HS to the fine-tuning loss to increase attention on key visual tokens and decrease attention on irrelevant tokens.For prompt-relevant tokens, entropy minimization reinforces their probabilities; for irrelevant tokens, the self-information term suppresses noisy pairs.
3. EXPERIMENT
Experiments evaluate ENCORE across benchmarks, cropping strategies, entropy temperatures, and ERT configurations. ENCORE improves accuracy broadly with 0.14% trainable parameters and modest inference overhead.
- Benchmark Results: ENCORE surpasses InternVL3-2B by 2.8% on TextVQA, 1.4% on OCRBench, 2.0% on RealWorldQA, and 2.4% on CCBench.The reported gains cover text-rich and non-OCR benchmarks.
- Benchmark Results: ENCORE reaches new highs on SEED-2 (65.5), InfoVQA (68.0), HallBench (43.1), POPE (90.2), and MMBench (79.2).The comparison is against 2B-parameter baselines.
- Efficiency: 0.14% of model parameters are trainable, while ENCORE increases FLOPs by 7.2% and inference latency by 10%.The reported efficiency results indicate modest computational overhead alongside accuracy gains.
- ECS Ablation: Model performance peaks at τ = 1, while increasing τ from 0.1 to 50 makes prompt-relevant token distributions more uniform and raises HS.The first or second layer performs comparably, whereas deeper layers show a marked drop.
- Cropping Strategy Comparison: ECS performs consistently better than alternative cropping methods on complex visual-semantic benchmarks such as SEED-2 and RWQA.MSAC performs comparably on OCR-focused datasets but degrades on more complex visual semantics.
- ERT Ablation: ERT is most effective in initial layers with subsequent layers frozen, and it concentrates attention on key visual tokens compared with dispersed attention without ERT.Small τ values sharpen prompt-relevant visual-token distributions, while deeper-layer fine-tuning underperforms standard NTP.
4. CONCLUSION
ENCORE addresses semantic fragmentation in lightweight VLMs through entropy-guided cropping and training that preserve prompt-relevant regions and sharpen key visual token extraction. Experiments report state-of-the-art performance among 2B-parameter VLMs.
- Conclusion: ENCORE preserves prompt-relevant regions and sharpens key visual token extraction through an entropy-guided framework.The framework targets semantic fragmentation in lightweight VLMs.