PaperlayerView exact revision on arXiv

Source-linked AI summary

SEER: Long-Context Reasoning via Selective Visual-Text Compression

Jiawei Xu, Zhilin Zhai, Jinrui Fang, Ruohan Xu, Mingfei Lu, Yi Zhang, Guanchu Wang, Tianlong Chen, Ying Ding

arXiv:2608.15962v1cs.CLcs.CV

TL;DR

Long-context reasoning is expensive, and uniform visual-text compression can lose details needed for precise extraction. SEER selectively identifies relevant images and retrieves source text only where needed, achieving 51.11% LongBench accuracy, 2.33 points above Glyph-9B.

  • Problem

    Uniform visual-text compression trades efficiency against precision because it compresses all context regardless of query relevance.

  • Method

    SEER learns a select-retrieve-reason pipeline that selects query-relevant images and retrieves their source text for reasoning.

  • Results

    51.11% average LongBench accuracy, 2.33 points above Glyph-9B and 3.49 points above Qwen3-8B.

  • Takeaways & Limitations

    Selective text retrieval improves precision on extraction-heavy tasks while retaining average prompt-token savings relative to full-text baselines.

  • Takeaways & Limitations

    Token savings are not guaranteed: SEER’s prompt is larger than the full-text prompt on seven of 21 tasks, and latency is higher at batch size 1.

Abstract

from arXiv · show

Long-context reasoning remains computationally expensive for large language models due to the quadratic complexity of attention over text tokens. Visual-text compression offers a promising alternative by rendering text into images and processing them with vision-language models, often reducing token usage. However, existing approaches apply uniform compression regardless of query relevance, potentially sacrificing precision where detailed extraction is required. We present SEER, a framework that learns to select query-relevant images through visual scanning and retrieve textual content only where needed, combining the efficiency of visual compression with the precision of text-based reasoning. Through supervised fine-tuning on tool-interaction trajectories, SEER learns adaptive tool invocation for selection and retrieval. Experiments on long-context benchmarks show that SEER improves extraction precision through selective text retrieval while retaining average prompt-token savings relative to full-text baselines. On LongBench, SEER achieves 51.11% average accuracy, outperforming the visual-text baseline Glyph-9B by 2.33 points and Qwen3-8B by 3.49 points. Code can be accessed at https://github.com/jiaweixu98/SEER

1 Introduction

SEER addresses the efficiency–precision trade-off in long-context reasoning by selectively retrieving text from query-relevant visual representations. Its three-stage, jointly trained pipeline improves LongBench accuracy over visual-text and full-text baselines while retaining visual compression benefits.

  • Motivation: Long-context processing is computationally expensive because self-attention has quadratic complexity in sequence length.This motivates more efficient approaches for applications including multi-turn dialogue and long-horizon tasks.
  • Motivation: Visual-text compression renders text as images for vision-language models, typically achieving 3–4× token compression while maintaining competitive accuracy.The compression ratio is measured as textual tokens relative to vision tokens.
  • Motivation: Uniform visual-text compression ignores query relevance, forcing a trade-off between efficiency from aggressive compression and detail preservation from conservative compression.Aggressive compression can lose fine-grained details needed for precise reasoning, whereas conservative compression reduces efficiency gains.
  • SEER: SEER selectively retrieves text only from images most relevant to inference, combining visual-text compression with selective tool invocation.The framework is designed to improve extraction precision without retrieving the full context text.
  • SEER: SEER uses a three-stage inference pipeline with training that jointly optimizes selection and reasoning capabilities.This training strategy teaches adaptive selection and reasoning within the same framework.
  • Results: 51.11% average accuracy on LongBench places SEER 2.33 points above Glyph-9B and 3.49 points above Qwen3-8B.Selective text retrieval yields gains of +8.75 on Qasper and +19.17 on MultiFieldQA-Zh.

2 Related Work

Related work explores visual-text compression for efficient long-context reasoning and tool-integrated architectures for complex problem solving. These approaches include image-based text processing, slow-fast visual/LLM pathways, OCR compression, and explicit stepwise reasoning with external tools.

  • Visual-Text Compression: Visual-text compression renders long texts into images for vision-language models, aiming to improve long-context reasoning efficiency.Glyph and related methods are presented as recent work in this direction, although the supplied passage truncates Glyph’s result.
  • Visual-Text Compression: Vist uses a slow-fast framework that skims low-salience context with images and a lightweight vision encoder while processing proximal tokens through the LLM.The fast path targets distant tokens, whereas the slow path supports fine-grained reasoning on proximal tokens.
  • Visual-Text Compression: 97% OCR precision is achieved by DeepSeek-OCR when the compression ratio remains below 10×.This condition means text tokens are within 10 times the number of vision tokens.
  • Tool-integrated Reasoning: CoT prompting established explicit decomposition into stepwise reasoning traces for complex problem solving.Each rationale incrementally builds upon previous steps, forming a foundation for tool-integrated reasoning architectures.
  • Tool-integrated Reasoning: Tool-integrated reasoning architectures enhance model capabilities through external tool integration, building on CoT prompting.The supplied passage identifies IRCoT as a prompting-based example.

3 Problem Setup

The problem is long-context question answering with visual-text compression, where a VLM answers from rendered context images and query-relevant source text may be selectively retrieved. This setup targets accurate answers while avoiding full-text decoding and exploiting the lower token cost of visual representations.

  • Given textual context C and query q, the task is to produce an answer y.
  • Visual-Text Compression: Visual-text compression renders context C into image segments I_j, which a VLM processes together with query q.
  • Limitations of Uniform Compression: Uniform compression processes all visual content regardless of query relevance, limiting granularity for precise extraction from specific images.SEER instead selectively retrieves source text from query-relevant images before final answer generation, avoiding full-text decoding.
  • Visual-Text Compression: 3–4× compression ratios τ_text/τ_vision have been reported while maintaining competitive performance.τ_text counts context text tokens, whereas τ_vision counts vision-encoder tokens over the rendered images.

4 Method

SEER combines visual selection, deterministic text retrieval, and reasoning in a unified tool-interaction trajectory. It is trained with supervised fine-tuning on complete Select-Retrieve-Reason trajectories, initialized from Glyph-9B.

  • Method Overview: SEER selects query-relevant images, retrieves their corresponding source text, and reasons over the combined visual-text representation.The framework begins by rendering textual context into images, then uses selected images to guide precise textual retrieval.
  • Training: SEER is trained with supervised fine-tuning on complete Select-Retrieve-Reason trajectories and initialized from Glyph-9B.Training supervises model-generated content with per-token weights while assigning zero weight to non-generated observation tokens.
  • Stage 1: Visual Selection: Visual selection predicts a set of valid image indices for rapid, query-relevant region selection.Duplicate indices are removed, valid indices are retained, and out-of-range indices are discarded.
  • Stage 2: Text Retrieval: Text retrieval is a deterministic lookup of source text corresponding to the selected images, enabling fine-grained local reasoning without OCR inference.The design focuses on learning when precise text is needed and can be replaced with OCR when source text is unavailable.
  • Stage 3: Reasoning: During reasoning, the model retains all visual images for global context while incorporating retrieved text for precise local information.The interaction is implemented as a structured tool call followed by a text observation and continued reasoning in a ReAct-style Thought-Action-Observation process.
  • Training Data Construction: Training trajectories combine rendered image sequences, queries, selected-image tool calls, retrieved-text observations, chain-of-thought reasoning, and final answers.The data is constructed from train splits of 2WikiMQA, HotpotQA, GovReport, LCC-Python, MuSiQue, Qasper, and TriviaQA.

5 Experiments

SEER is evaluated on LongBench against text-based long-context models and visual-text compression baselines, with Glyph-9B as the primary controlled comparison. The experiments specify the benchmark composition, training setup, and official metric aggregation used for comparison.

  • Datasets: 21 tasks across 6 categories and 4,750 evaluation samples define the LongBench evaluation setting.Average context length is 6,711 words for English and 13,386 characters for Chinese.
  • Baselines: Three baseline groups are reported, with Glyph-9B serving as the primary controlled comparison and external text-based LLMs providing practical context.Baselines prioritize open-weight models of roughly 7B–9B parameters with similar release periods.
  • Implementation Details: SEER is initialized from Glyph-9B and fine-tuned on Select-Retrieve-Reason trajectories for 6,000 optimizer steps.Training uses 2× H100 80GB GPUs, learning rate 3 × 10^-6, global batch size 2, and maximum sequence length 16,384.
  • Evaluation Metrics: LongBench evaluation follows the official protocol with dataset-specific metrics for QA, summarization, classification, retrieval, passage count, and code tasks.The final Avg/Overall score is computed as a macro-average over the benchmark categories.

6 Results

SEER improves LongBench accuracy over Glyph-9B and Qwen3-8B by selectively retrieving text for query-relevant visual content. This gains extraction precision while retaining prompt-token savings, with retrieval serving as the key contributor to performance.

  • Main results: 51.11% LongBench Avg surpasses Glyph-9B by 2.33 points and Qwen3-8B by 3.49 points.Glyph-9B scores 48.78%, while Qwen3-8B scores 47.62% under the reported setup.
  • Extraction-heavy tasks: +8.75 on Qasper, +4.22 on NarrativeQA, and +3.80 on HotpotQA show gains on extraction-heavy tasks.SEER improves Qasper from 38.99 to 47.74, NarrativeQA from 21.83 to 26.05, and HotpotQA from 56.42 to 60.22.
  • Efficiency: 7,623.8 average prompt tokens gives SEER 1.28× compression, between GLM-4-9B-Chat-1M at 10,175.8 and Glyph-9B at 3,522.5.Selective retrieval sacrifices some maximal compression to recover task-relevant textual precision.
  • Selective retrieval behavior: 1.59 valid images per query on average, with fewer than 1% selecting more than three, indicates sparse visual selection despite 6.6 rendered pages per sample.At an intermediate retrieval budget, k=1 already recovers +3.00 of the unweighted-over-tasks k=0→k=all gap of +4.88.
  • Ablations: 51.11% with full retrieval versus 43.00% without retrieval demonstrates an +8.11 category-level macro-average gain from the complete pipeline.The selection-only variant obtains 43.00% LongBench Avg, while the pure-visual variant obtains 48.78%; an iso-training control reaches 45.20.

7 Conclusion

SEER improves visual-text compression by selecting query-relevant pages and retrieving exact source text only where needed. On LongBench, it reaches 51.11% overall accuracy, surpassing Glyph-9B by 2.33 points.

  • 7 Conclusion: SEER reads the rendered document, identifies query-dependent pages, and retrieves exact source text only from those pages.This query-selective process addresses the precision costs of treating every page alike.
  • 7 Conclusion: 51.11% overall accuracy on LongBench puts SEER 2.33 points above Glyph-9B.The gains are concentrated on extraction-heavy tasks.

Limitations

SEER’s results are bounded by unreliable selection outputs and narrower-than-suggested efficiency gains. The model can emit invalid page indices, while prompt-token savings are averages rather than guarantees.

  • Selection robustness: SEER occasionally emits out-of-range page indices, which are discarded and could cause retrieval failures if trusted in deployment.The limitation concerns selection as a deployment interface.
  • Efficiency: On seven of 21 tasks, SEER’s prompt is larger than the full-text prompt, showing that token savings are not guaranteed.The efficiency claim is narrower than the compression ratio suggests because savings are reported as an average.

A Appendix · A.1 Implementation Details · A.1.1 Model Architecture

SEER builds on Glyph-9B, a GLM-4V-based model comprising a vision encoder and language-model backbone, with detailed architecture specifications provided in Table 5.

  • A.1.1 Model Architecture: SEER builds upon Glyph-9B, which is based on the GLM-4V architecture.The implementation uses Glm4vForConditionalGeneration.
  • A.1.1 Model Architecture: The model consists of a vision encoder and a language model backbone.
  • A.1.1 Model Architecture: Table 5 provides the model architecture specifications.

A.1.2 Special Tokens … A.13 Qualitative Success and Failure Cases

The appendix specifies SEER’s tool-interaction format, training-data construction, ablations, efficiency measurements, selector analyses, model comparisons, and qualitative strengths and limitations. Across these evaluations, SEER’s mechanism improves targeted extraction but remains challenged by distributed multi-hop evidence.

  • A.4 Ablation Protocol: Without Retrieval (w/o Retrieve): k=0 removes retrieved text while retaining selection, testing whether SEER’s benefit comes from identifying relevant images or receiving precise textual content.The same trained model is used, and inference only changes the observation supplied after the tool call.
  • A.5 Remaining LongBench Task Results; A.6 Per-Task Token Breakdown; A.7 Wall-Clock Efficiency and Long-Context Scaling: SEER’s efficiency advantage is context capacity rather than single-request LongBench latency: visual compression saves approximately 67% KV memory, while text prefill rises from 0.26 s at 5.4k tokens to 5.53 s at 79k.At LongBench lengths, prefill is comparable, decode throughput is approximately 76 tok/s, and visual inputs extend feasible context length beyond roughly 128k tokens.
  • A.8 Iso-Training Control: Glyph-9B-SFT-direct: 52.54 on seven QA tasks shows SEER outperforming both Glyph-9B at 45.98 and direct fine-tuning at 44.67, while the k=0 condition reaches 43.00.Direct fine-tuning falls on few-shot tasks, including LSHT 45.50 →22.25 and SAMSum 32.82 →9.60, whereas retaining the mechanism avoids this degradation.
  • A.9 Intrinsic Page-Selection Quality: 0.578 F1 for SEER’s learned selector exceeds random match-k at 0.413 and first-page match-k at 0.486, while BM25 is slightly higher at 0.606 on the gold-nonempty subset.The evaluation re-annotates 4,750 examples containing 31,182 rendered pages, with baselines matched to SEER’s selected-page budget.
  • A.10 End-to-End Comparison with a BM25 Page Selector; A.11 Comparison with Recursive Language Models: 48.34 versus 44.03 over 15 matched tasks shows SEER selection outperforming BM25-forced selection under the same answering pipeline, while RLM-Qwen3-8B reaches 42.60 versus SEER’s 52.54 on seven QA tasks.BM25 remains competitive in lexical regimes, whereas SEER gains on multi-hop and layout-sensitive tasks, including MuSiQue +10.1, 2WikiMQA +11.4, and MultiFieldQA-Zh +6.1.
  • A.12 Task-Wise Retrieval Heterogeneity Analysis; A.13 Qualitative Success and Failure Cases: SEER succeeds when exact lists, numbers, or polarity are localized on one or two pages, but fails when multi-hop evidence is distributed across pages and must be chained beyond one selection round.Observed failures include answering an intermediate MuSiQue hop without reaching the final entity and resolving a role chain to the wrong person.
Loading 2608.15962v1…