Source-linked AI summary
Scalable Visual Pretraining for Language Intelligence
Yiming Zhang, Zhonghan Zhao, Wenwei Zhang, Haiteng Zhao, Tianyang Lin, Huanze Tang, Yunhua Zhou, Demin Song, Kuikun Liu, Haochen Ye, Haian Huang, Yuzhe Gu, Haijun Lv, Qipeng Guo, Bin Liu, Gaoang Wang, Kai Chen
TL;DR
Text-only pretraining can lose information conveyed by figures, equations, tables, and layouts in visual documents. The paper studies Visual Pretraining, which learns from raw document visuals without text extraction or paired image-text supervision, and finds consistent gains over matched text pretraining across backbones and benchmarks. These results position visual pretraining as a scalable complement to language pretraining, though the method still depends on text pretraining.
Problem
Textualizing visual documents can discard reasoning-relevant information in figures, equations, tables, and layouts, limiting how language models learn from scientific knowledge.
Method
Visual Pretraining adds next visual-latent prediction on native document pages to continued text pretraining, without text extraction or image-text pairing supervision.
Results
Visual Pretraining consistently outperforms matched text pretraining across multiple backbones and scientific reasoning benchmarks, using only 25% of the token budget.
Takeaways & Limitations
Native visual representations provide useful supervision beyond textualized content and can improve reasoning, visual perception, and cross-modal alignment without labeled cross-modal data.
Takeaways & Limitations
Visual Pretraining is not independent of language pretraining because it introduces visual-latent prediction during continued pretraining grounded in text pretraining.
Abstract
from arXiv · showhide
The rapid progress of large foundation models has been driven predominantly by pretraining on large-scale text corpora. However, many forms of knowledge are conveyed through visual representations, where figures, typeset equations, and page layouts carry rich information that cannot be faithfully or completely captured by text alone. Yet current pretraining approaches discard these visual cues by converting visually rich sources, such as documents and web pages, into plain text for learning language intelligence. This paper challenges the default assumption that language models must be trained on text-only representations and shows that Visual Pretraining is a scalable learner for foundation model intelligence. To this end, we conduct a systematic study of unsupervised visual pretraining paradigms that directly leverage visual documents without text extraction. Across multiple backbones and benchmarks, visual pretraining on the same underlying corpora consistently outperforms text-only pretraining, offering an efficient pathway to scalable language intelligence.
1. Introduction
Visual documents contain reasoning-relevant information in diagrams, equations, tables, and layout that text-only pretraining can discard. The paper introduces visual pretraining to learn directly from raw documents and reports advantages over matched text pretraining.
- Motivation: Plain-text conversion can discard or distort diagrams, equations, tables, and page-level structure encoded in scientific documents.These visual cues can make spatial, symbolic, and structural relations available for reasoning.
- Motivation: Existing text and multimodal regimes leave the raw visual modality outside the model’s learning objective.Documents are typically parsed into text, while visual inputs in multimodal training condition text-token prediction.
- Approach: Visual Pretraining learns directly from raw documents without text extraction or image-text pairing supervision.The framework trains a foundation model using document-native visual information rather than first converting pages into text.
- Findings: VP consistently outperforms text-only pretraining across multiple LLM backbones and scientific reasoning benchmarks on matched corpora.The comparison uses the same underlying corpus, isolating document representation as the controlled difference.
2. Results
Across matched scientific-document corpora, Visual Pretraining (VP) improves text-only reasoning, scales efficiently with visual tokens, and transfers to multimodal alignment and visual reasoning. Its strongest advantages appear where figures, equations, tables, and layouts carry substantial evidence.
- Effectiveness: VP consistently improves scientific reasoning over matched text pretraining across multimodal and language-only backbones.The two settings differ only in document representation, using the same sources and supervised fine-tuning stage.
- Effectiveness: GPQA Diamond improves by up to 3.22 points and MMLU-Pro by up to 2.1 points across four backbones, while HLE gains reach only 0.97 points.The paper attributes the larger gains to visual content in scientific documents and describes HLE as primarily hard multi-step reasoning.
- Scalability: VP reaches a lower final SFT loss and converges faster than TP after comparable continued-pretraining loss.The authors note that SFT loss is more closely grounded in downstream-relevant geometric structure than pretraining loss.
- Efficiency: 1.27× on MMLU-Pro, 2.02× on GPQA, and 2.88× on AIME-25 are VP’s normalized gains using approximately 20B visual tokens versus roughly 80B text tokens.The gains increase as training progresses on the same PDF corpus.
- Efficiency: The 1× setting retains 8,192 foreground visual tokens per batch, while lowering maximum resolution preserves strong performance at that fixed visual budget.The 1× configuration offers the best trade-off under a fixed training context, and VP still outperforms TP after resolution reduction.
- Cross-modality: VP improves cross-modal alignment and multimodal benchmark performance without image–text pair supervision.On 100 image–text pairs, centroid separation drops from 1.665 to 0.661 and paired cosine similarity rises from 0.631 to 0.907; VP outperforms base and TP across reported multimodal benchmarks.
- Visual grounding: VP’s advantage over TP grows with visual-structure density and reaches +5.4 on ChartQAPro for both backbones and +4.8 on MathVista for Llama 3.2 Vision.Low-density, text-dominant pages produce comparable VP and TP performance, whereas visually heavy content yields larger improvements.
3. Discussion
Visual Pretraining (VP) preserves document-native visual structure while complementing text continued pretraining, improving scientific reasoning and cross-modal capabilities without paired image-text supervision. The evidence supports VP as a compact, promising pretraining route, but its scope remains centered on scientific PDFs and its relationship to language pretraining is complementary rather than independent.
- Evidence and mechanism: Visual-token reasoning attends to semantically relevant regions of rendered problems, including constraints and intermediate computation steps, similarly to textual attention patterns.Figure 4 compares attention from answer regions to retained page patches against attention from final answer sentences to preceding text tokens.
- Evidence and mechanism: VP improves scientific reasoning over matched text pretraining across backbones, with larger multimodal gains on visually dense examples.The paper presents this as evidence that visual elements can carry reasoning-relevant knowledge, while noting that attention analysis is supportive rather than mechanistic proof.
- Core interpretation: VP mixes text continued pretraining with next-visual-latent prediction on native scientific pages, preserving layout, formulas, diagrams, and other visual relations.The visual objective models foreground visual tokens ordered by page position, rather than treating pages solely as textualized content.
- Scope and limitations: VP is best understood as an extension of language pretraining, because the reported experiments ground visual-latent prediction in text pretraining rather than using a language-independent visual learner.The authors identify better coordination between visual-latent and text decoding as a future direction.
- Scope and limitations: The experiments primarily target high-knowledge-density scientific PDFs, leaving transfer to broader visual corpora such as natural images or video open.The limitation concerns whether the same pretraining strategy generalizes beyond pages whose visual layouts, figures, tables, formulas, and text are tightly coupled.
4. Methods
VP extends text continued pretraining with a sparse visual sequence derived from raw scientific pages and trains the model to predict the next visual latent alongside text tokens. The method preserves foreground content and page order while using a frozen vision tower, matched PDF sources, and a compact visual token stream.
- Visual representation: Each rendered page is encoded by a frozen vision tower, filtered to retain foreground patches, projected into the LLM hidden space, and ordered in raster-scan order.Blank background regions are removed while positional encodings preserve spatial layout.
- Visual representation: The sparse visual representation keeps foreground document content in page order while substantially shortening the visual context.The retained sequence has length L much smaller than the original feature sequence N.
- Visual objective: For each document image, the LLM predicts the next frozen visual latent from its current hidden state under a causal attention mask.An output projection head maps the hidden state back to the frozen visual-latent space for next-feature prediction.
- Visual objective: The visual loss matches each predicted feature to its correct next target using cosine-similarity probabilities with temperature while treating other batch features as negatives.This expanded InfoNCE objective distinguishes the correct next document feature from other patches.
- Joint training: The final objective combines autoregressive text cross-entropy with next-visual-latent prediction, and text and visual examples are interleaved at a fixed mixing ratio.The LLM, visual input projection, and prediction head are updated while the visual encoder remains frozen.
- Controlled comparison: TP and VP use the same non-PDF corpus, starting checkpoint, optimization recipe, SFT stage, and underlying scientific PDFs, differing only in PDF representation.TP uses approximately 80B parsed text tokens, whereas VP uses approximately 20B sparse visual tokens, producing 180B versus 120B total CPT tokens.
- Evaluation: Evaluation uses zero-shot benchmarks after SFT initialized from continued pretraining, with chain-of-thought prompting for Table 1 and direct-answer templates for Table 2.Reported metrics include GPQA pass@8, AIME-25 averages over 32 runs, and pass@1 for MMLU-Pro and HLE unless otherwise specified.
A. Detailed Visual-Pretraining Pipeline
The visual-pretraining pipeline converts raw document pages into sparse, ordered visual sequences and trains a shared autoregressive LLM to predict successive visual features. It combines visual prediction with text training while keeping the vision encoder and spatial merger fixed.
- Visual feature construction: A frozen ViT encodes each rendered document page into patch features, while a frozen spatial merger produces the visual prediction targets.The pipeline uses fixed visual representations rather than reconstructing pixels.
- Visual feature construction: Foreground filtering removes blank patches, retains merged tokens containing foreground content, and orders the retained features in raster-scan order.This produces a sparse sequence that preserves page order while removing most blank regions.
- Sequence formation: The retained features are projected into the LLM hidden space, assigned raster-order positions, and processed as an ordered visual sequence.Position indices follow the raster order after background removal, avoiding allocation to discarded regions.
- Causal visual prediction: With causal attention over visual positions, each hidden state predicts the next foreground feature through a lightweight MLP head.The target is the frozen next visual feature, making the objective a continuous analogue of next-token prediction.
- Training objective: The visual loss uses cosine-similarity logits, batch negatives, and a softmax matching probability, alongside standard text next-token prediction.Only the LLM, visual input projection, and prediction head are updated; the ViT and spatial merger remain frozen.
- Training efficiency: Packed document sequences use block-causal masking so visual tokens cannot attend to future tokens, other pages, or targets across document boundaries.This improves training efficiency while restricting prediction targets to each original foreground sequence.
B. PPL-Based Image-to-Text Retrieval
The paper evaluates cross-modal alignment by matching document images to candidate texts using conditional-perplexity scores. Visual pretraining substantially improves retrieval quality and reduces spurious text-image associations.
- Evaluation: The evaluation samples 100 document image–text pairs and scores every image-text combination using a conditional-perplexity PMI-style measure.Lower scores indicate stronger alignment, and performance is reported with Recall@K and Mean Reciprocal Rank.
- Results: R@1 increases from 64.0% to 99.0% and MRR rises from 78.2 to 99.5 after visual pretraining.The result is reported for the 100 × 100 image-text score matrix.
- Results: Off-diagonal score variance drops from σ=0.168 to σ=0.043, eliminating hub attractors and enabling near-perfect discrimination.In the original model, some texts receive spuriously low scores regardless of the input image.
C. Visual Pretraining with Generative Decoder
The generative-decoder variant adds pixel-level reconstruction supervision by decoding visual latents back to pixels, whereas the decoder-free variant predicts foreground patch latents directly. The added supervision provides limited reasoning benefit while increasing computational cost, motivating the simpler formulation.
- Architecture: The generative-decoder variant adds a MAR decoder and frozen VAE decoder to reconstruct pages from LLM-refined latents.This introduces pixel-level supervision absent from decoder-free latent prediction.
- Cost: The generative-decoder design adds approximately 300M trainable parameters, requires both diffusion-style latent and pixel MSE losses, and reduces throughput by roughly 30%–40%.It requires approximately 1.4× the training time to reach the same convergence criterion.
- Downstream reasoning: Both visual-pretraining variants outperform the Base model and text-only baseline across evaluated reasoning benchmarks, while pixel-level loss yields only comparable or marginally better scores.The comparison uses matched document sources and the same SFT protocol.
- Architecture: The decoder-free formulation autoregressively predicts foreground patch latents without pixel reconstruction.It avoids the additional reconstruction decoder used by the generative variant.
- Conclusion: The paper adopts decoder-free visual pretraining because it achieves comparable downstream performance through a substantially simpler and more efficient pipeline.The conclusion characterizes pixel-level generation supervision as offering limited marginal benefit for language reasoning.
D. Additional Implementation and Evaluation Details
The implementation matches visual and text pretraining on the same PDF sources while differing in document representation, then evaluates training cost, reasoning, and cross-modal alignment. VP predicts visual features directly through a decoder-free pathway and improves alignment across global, structural, and local metrics.
- Data construction: Matched PDF pages provide approximately 20B retained visual tokens for VP versus approximately 80B parsed text tokens for TP.This construction controls the document source and changes only the representation of the scientific-PDF corpus.
- Model architecture: VP uses the corresponding LLM backbone with a frozen ViT-style vision tower, trainable visual projector and LLM, and a prediction head for next visual latent prediction.Foreground visual features are mapped into the LLM hidden space and predicted in the frozen visual-feature space.
- Model architecture: Decoder-free VP predicts frozen visual features rather than reconstructing pixels, avoiding the computational cost and optimization complexity of a pixel-level image decoder.A separate generative-decoder variant is studied independently.
- Optimization: Visual and text examples are interleaved during continued pretraining, with autoregressive cross-entropy for text and an InfoNCE next-visual-latent loss for VP.The main Qwen3.5 runs use temperature τ=0.07 and VP loss weight 0.1.
- Evaluation: The alignment analysis uses 100 matched document image–text pairs and compares pooled last-layer visual and textual embeddings before and after VP.Alignment is assessed through global distance, structural geometry, and local neighbourhood consistency.
- Alignment results: Centroid separation drops 60% from 1.665 to 0.661, pairwise cosine similarity rises from 0.631 to 0.907, and linear CKA increases from 0.657 to 0.745 after VP.Mutual k-NN overlap also improves at k=1, 5, and 10, with the largest gain at k=1.