Source-linked AI summary
ZeroSense:How Vision matters in Long Context Compression
Yonghan Gao, Zehong Chen, Lijian Xu, Jingzhi Chen, Jingwei Guan, Xingyu Zeng
TL;DR
Existing VTC evaluations can overestimate text preservation because downstream models use linguistic priors to infer missing content. The paper introduces a decoupled framework and ZeroSense benchmark to isolate preserved text from semantic inference and raw OCR ability, finding that VTC quality and downstream accuracy diverge substantially.
Problem
Existing downstream-task evaluations do not reliably measure text preserved by visual-text compression because semantic coherence lets MLLMs infer missing text.
Method
The paper decouples semantic priors, raw OCR capability, and compression quality using a performance framework and the semantically independent ZeroSense benchmark.
Results
VTC quality and downstream task accuracy diverge significantly across datasets, including 81.3% end-to-end accuracy versus 27.4% Kquality on Fox at 17.5× compression.
Takeaways & Limitations
Reliable VTC evaluation requires semantic decoupling so results reflect visual-text preservation rather than downstream models’ semantic inference.
Takeaways & Limitations
The framework’s absolute inference likelihood remains bounded by the OCR capabilities of the evaluated MLLMs.
Abstract
from arXiv · showhide
Recent visual-text compression (VTC) methods, typified by DeepSeek-OCR, report impressive high token compression ratios for long-context modeling tasks by leveraging text-to-image rendering. However, existing evaluation protocols heavily rely on downstream task performance. Such evaluation metrics fail to accurately measure text preservation due to the strong inherent linguistic priors of Multimodal Large Language Models (MLLMs). In this work, we introduce a new evaluation framework that decouples MLLMs' capabilities to faithfully assess VTC quality. Within this framework, we further introduce the ZeroSense Benchmark to ensure low semantic correlation of testing samples. By eliminating contextual dependencies, our benchmark guarantees that the evaluation results are purely reflective of VTC quality, unaffected by the semantic inference capabilities of downstream models. Extensive experiments across multiple datasets demonstrate that VTC quality and downstream task accuracy diverge significantly, highlighting the necessity of our decoupled evaluation framework.
1 Introduction
Visual-text compression replaces many textual tokens with compact document images, but downstream performance can conceal text loss because MLLMs use contextual inference and have raw perceptual limits.
- Visual-text compression renders extensive textual sequences into compact document images, replacing thousands of textual tokens with visual tokens.The paradigm targets long-context bottlenecks caused by the quadratic complexity of self-attention.
- DeepSeek-OCR can correct typographical errors through contextual inference, obscuring text loss incurred during compression.This semantic-prior compensation makes end-to-end performance an incomplete indicator of preserved text.
- Incoherent alphanumeric strings can produce glyph-similar errors even at high resolution, exposing a raw perceptual bottleneck.The failure occurs when semantic structure is weak and contextual inference cannot compensate reliably.
2 Related Work
Document understanding has moved from OCR cascades toward unified visual and OCR-free models, while compression improves efficiency but makes text preservation difficult to measure reliably.
- Early document-understanding systems used OCR cascades that discarded layout cues and suffered from error propagation.These methods were effective for text-dominant documents but separated text extraction from later language-model processing.
- Unified end-to-end and OCR-free paradigms jointly model document content, layout, and appearance or predict structured outputs directly from images.These approaches use encoder–decoder OCR, document-intelligence systems, or image-to-text pretraining.
- Compression methods improve scalability by reducing document pages to fixed visual-token budgets, but can obscure how much text is preserved rather than reconstructed from semantic priors.Existing evaluations often rely on question answering, where linguistic priors can conceal compression-induced text loss.
- Common OCR and document benchmarks have limitations for high-ratio optical compression, including sparse-text scene bias and strong semantic coherence.Semantic coherence lets language models guess missing text, masking visual distortion caused by extreme compression.
- ZeroSense preserves realistic full-page layouts while eliminating semantic priors, enabling controlled evaluation of optical fidelity.Its semantic vacuum separates compression quality from linguistic inference effects.
3 Framework
The framework measures visual-text compression by comparing compressed visual inputs with pure-text inputs while separating semantic priors, raw OCR ability, and preserved text through ZeroSense.
- 3.1 Formulation: Compression ratio ρ(θ) is defined as original text-sequence length divided by the aggregate visual-token count across generated frames.The metric captures the number of original textual tokens encoded per visual token.
- 3.1 Formulation: The framework evaluates VTC by quantifying performance discrepancies between visually compressed inputs and pure-text inputs across compression ratios.Its protocol requires the measured performance delta to be invariant to the semantic content of the source text.
- 3.1 Formulation: The framework maximizes performance across a predefined model set to separate compression quality from the evaluated MLLM’s OCR capability.Absolute inference likelihood is bounded by the model’s visual recognition ability, so a model-set maximum provides the theoretical upper bound considered by the protocol.
- 3.2 OCR Formulation Task: For OCR restoration, the target output becomes the original text, shifting evaluation toward the relationship between compression ratio and compressed-versus-text performance.The pure-text condition has theoretically 100% probability for the original text, simplifying the comparison.
- 3.2 OCR Formulation Task: End-to-end recognition combines contextual prior guessing, raw single-page OCR capability, and the text-preservation quality Kquality of the rendering strategy.The framework uses ZeroSense to isolate the prior component and independently measures raw OCR capability.
- 3.2 OCR Formulation Task: The derived evaluation measures text preservation independently of the base model’s reasoning bias and raw recognition bounds.A calibrated accuracy baseline is also proposed to account for inherent MLLM OCR perceptual ability.
- 3.3 Decoupling Semantic Priors and OCR: ZeroSense uses non-semantic-correlated character sequences with identical rendering parameters to quantify semantic-prior effects.Its downstream performance represents the semantic-vacuum condition used to estimate contextual compensation.
4 ZeroSense Benchmark
ZeroSense constructs an auxiliary benchmark with method-specific rendering layouts and semantically irrelevant visual content. Its pipeline extracts layout features, estimates typography and capacity, then generates contextually irrelevant text for rendering.
- Benchmark construction: ZeroSense provides precise rendering layouts while guaranteeing semantic irrelevance in the visual content.The benchmark specifies features such as font size, case, capacity, and line height for method-specific evaluation.
- Pipeline: The Analyzer extracts visual attributes from input images using OCR results or ground-truth annotations.Extracted attributes include word counts and bounding-box coordinates.
- Pipeline: The Renderer combines extracted visual attributes with generated text to synthesize final images.The Text Generator produces semantically irrelevant text before rendering.
- Feature extraction: Font size Sopt is estimated as the maximum integer size whose rendered text fits the target block dimensions.The method uses a rendering simulation and iterative search to match the block height and width.
- Feature extraction: Text capacity Ci is counted in words for Latin scripts and characters for logographic systems.A language-aware counting strategy uses language detection based on ASCII density.
- Semantically irrelevant sample generation: The replacement algorithm samples low-probability valid tokens to disrupt contextual predictability.This creates a semantic vacuum intended to make evaluation depend on visual retention rather than language-model priors.
5 Experiment
Experiments show that downstream accuracy and visual-text preservation diverge because semantic priors can compensate for degraded visual input. The decoupled metrics expose this divergence across datasets and compression ratios.
- Experimental Setup: The evaluation uses Fox, Omni, DI-100 v1.3, and ZeroSense to measure document understanding, end-to-end performance, and isolated text preservation.Character-level precision and normalized edit distance operationalize generation quality, while ZeroSense separates visual fidelity from semantic inference.
- Text Preservation: 81.3% accuracy at 17.5× compression on Fox contrasts with Kquality of 27.4%, revealing substantial visual degradation masked by contextual inference.Fox documents contain contextual dependencies that allow MLLMs to infer tokens despite visual information loss.
- Text Preservation: 97.1% Kquality at 7.5× compression on Omni exceeds its 89.2% standard end-to-end accuracy.This reversal demonstrates that downstream accuracy does not consistently track isolated visual-text preservation.
- Dataset Effects: At 17.5× compression, the framework–DeepSeek-OCR discrepancy is 53.9% on Fox versus 11.7% on Omni.The datasets differ in token distributions: Fox is concentrated, whereas Omni spans multiple text-density intervals, including very low- and high-density samples.
- Semantic Compensation: Semantic priors account for 23.8% of Fox predictions at 7.5× compression and 67% at 17.5×, while Omni reliance fluctuates between 31.7% and 45.3%.Prior reliance increases with compression on Fox, whereas Omni exhibits a broader fluctuation across compression levels.
- Raw Recognition: OCRraw declines from 39.5% to 17.4% on Omni and from 76.1% to 46% on Fox as compression rises from 7.5× to 17.5×.These calibrated measurements isolate raw visual extraction from semantic priors.
- Perturbation Validation: Shuffling word order yields only an 18.6% precision gap at 17.5× compression, insufficient to disrupt language-level regularities for evaluation.The experiment uses 1,478 images with five random permutations per image across four DeepSeek-OCR resolution modes.
6 Conclusion
The paper identifies semantic-prior coupling as a critical flaw in VTC evaluation and introduces a decoupled framework with ZeroSense. Experiments show that VTC quality and downstream accuracy can diverge significantly.
- Conclusion: Existing protocols couple text preservation with MLLMs’ linguistic priors, biasing assessments based on downstream task performance.The paper treats this coupling as a critical oversight in evaluating visual-text compression.
- Conclusion: The proposed framework and ZeroSense benchmark decouple visual-text preservation from semantic inference using low-semantic-correlation evaluation samples.The benchmark is designed so results reflect visual-text preservation rather than contextual guessing.
- Conclusion: Experiments reveal significant divergence between VTC quality and downstream task accuracy, motivating more reliable evaluation of long-context compression.The conclusion positions the framework as a foundation for assessing future long-context modeling architectures.
A Extended Evaluation Metrics: Normalized Edit Distance
The supplementary evaluation adds edit distance to measure how semantic relevance affects end-to-end outcomes. Across compression ratios from 7.5× to 17.5×, removing semantic priors substantially compromises model output performance, showing that inference relies on more than preserved text.
- Edit distance complements Fprior by measuring the impact of semantic priors on end-to-end outcomes.
- 7.5× to 17.5× compression ratios show significantly degraded output performance after semantic priors are eliminated across both datasets.
- The results indicate that models use strong semantic priors for inference rather than relying exclusively on text preservation.
- The negative sign in Supplementary Table 1 denotes performance degradation under non-semantic conditions, not the absolute metric difference.
B.2 Data Statistics of ZeroSense
ZeroSense statistics validate semantic decoupling while preserving document-layout characteristics across datasets. The analysis contrasts Fox’s regular token distributions with Omni’s extreme cases and documents the layout-extraction and rendering strategies used to reproduce baseline styles.
- Omni contains many extreme token-count cases, whereas Fox is concentrated in a more reasonable distribution.
- Omni’s extreme rendering cases can exceed OCR recognition capability, making extracted text-preservation and end-to-end values relatively close.
- The pipeline adapts rendering to source complexity by using normalized blank canvases for clean backgrounds and inpainting for complex layouts.
- ZeroSense replaces original semantic content with low-probability tokens while preserving document structural context.
- Layout reconstruction uses raw OCR boxes, vertical projection profiles, noise tolerance, spatial gaps, and box merging to recover paragraph-level structure.
- Font size and text capacity are derived from bounding-box dimensions and rendering simulations using language-aware calculations.
C.3 Semantically Irrelevant Sample Generation
ZeroSense generates semantically irrelevant text at layout-prescribed capacities so visual rendering can disrupt contextual predictability without changing the target document structure. Its causal-LM procedure samples from dynamically filtered low-probability vocabulary subsets.
- The generation algorithm replaces original text with semantically irrelevant content to disrupt contextual predictability.
- Qwen2.5-7B serves as a probability evaluator while tokens are sampled from a low-probability subset Qt at each generation step.
- Dynamic threshold relaxation is applied when the restricted vocabulary subset is empty, maintaining generation robustness.
- The procedure generates a sequence of target capacity Ci tokens from a valid vocabulary using posterior probabilities conditioned on prior context.
- The broader construction pipeline is designed to ensure reproducible ZeroSense benchmark generation from extracted layout features and semantic generation.
D.1 The Pipeline of the Perturbed Dataset Construction
The perturbed dataset alters document semantics while retaining visual layout and pixel-level statistics. It combines geometry-constrained line rearrangement with density-preserving word shuffling, producing images that visually resemble the originals despite scrambled word order.
- DDI-100 v1.3 supplies word-level coordinates that enable cropping and rearrangement during perturbation construction.
- The construction changes semantic content while largely preserving the original visual layout and pixel-level statistical properties.
- Inter-line rearrangement groups lines geometrically, limits within-group size variation to 5%, and randomly exchanges only lines in the same group.
- Intra-line word shuffling randomly reorders words while constraining spacing to preserve visual density.
- The resulting shuffled images have scrambled word sequences but nearly identical layout and font-size characteristics.