From Text to Pixel: Advancing Long-Context Understanding in MLLMs

• Long-form multimodal tasks combine multiple text-rich images with long-form text generation, requiring cross-image reasoning and context-grounded responses.Long inputs can exceed model context limits, while long outputs must remain coherent and avoid irrelevant or hallucinated content.Source: §1 · p. 2
• SEEKER represents textual information through visual tokens to convey more information within a fixed context length.The approach targets compact tokenization without additional computation overhead.Source: §1 · p. 2
• For an eight-page document, SEEKER uses 4,608 image tokens instead of roughly 10k OCR tokens, fitting within an 8k-token context limit.Each page is converted into 576 tokens before reasoning and generation.Source: §1 · p. 2
• SEEKER interleaves image tokens with textual data to preserve coherence across extended multimodal sequences and support reasoning across multiple images.Source: §1 · p. 2
• Instruction-tuned SEEKER achieves promising results against existing MLLMs on six long-context multimodal tasks.Source: §1 · p. 2

• Existing MLLMs include proprietary and open-source vision-language systems, but their long-context capabilities remain underexplored.Prior work has also identified deficits in reasoning over multiple images and explored pixel-based processing through task-specific fine-tuning.Source: §2 · p. 3
• The paper frames SEEKER as advancing long-context multimodal understanding through long-form image inputs and long-form text outputs.Source: §2 · p. 3
• Long-context Transformers face context-length challenges, while MLLMs add difficulties from vision encoders and connections between vision and language components.Related approaches include context extension, dynamic token allocation, and inference-time vision-model slimming.Source: §2 · p. 3
• SEEKER differs by using image tokens as compact representations for image and text, reducing the language model context required for equivalent semantic information.Source: §2 · p. 3

3.1 Using Image Tokens to Encode Text Helps Context Length Extrapolation

• Visual tokens improve context-length extrapolation as document inputs grow from approximately 1k to 500k text tokens.Text-input models show significant Rouge-L degradation, whereas document images improve extrapolation without additional changes.Source: §3.1 · p. 3

3.2 Long-Context Multimodal Task

• SEEKER evaluates long-context multimodal input as multiple text-rich images interleaved with text and long-form text output.The task design covers both understanding extended multimodal inputs and generating long responses.Source: §3 · p. 3, §3.2 · p. 4
• Training data combines multi-image reasoning, document-page instructions, image grounding, and verbatim reading tasks.Scientific PDFs provide 4–24 page image sequences, while generated instructions target description, conversation, grounding, and exact text reading.Source: §3.2 · p. 4

3.3 Long-Context Multimodal Large Language Model

• SEEKER interleaves image and text features with image separators, projects image features into the word-embedding space, and preserves image-specific identifiers.The sequence is represented as Q = [t0; v1; t1; v2; t2; . . . ; vn; tn], supporting image-specific reasoning across multiple images.Source: §3.3 · p. 5
• Dense image-text alignment renders Wikipedia paragraphs into images and increases text density gradually from sparse to dense representations.The training tasks include multi-image question answering and verbatim reading from rendered text images.Source: §3.3 · p. 5
• SEEKER uses autoregressive supervised fine-tuning to predict each next output token from preceding visual and textual tokens.The objective conditions target output tokens on multimodal queries Q and model parameters θ.Source: §3.3 · p. 5

4.1 Model Architecture

• SEEKER converts 1024 × 1024 images into a fixed 576-token representation using SigLIP-based vision encoders and a hybrid MLP adaptor.SEEKER uses a hybrid SigLIP-L and SAM-B encoder, while SEEKER-TINY uses SigLIP.Source: §4.1 · p. 5
• Table 2 reports accuracy or Rouge-L together with tokens per image for proprietary, proposed, and other models.The table covers long image and text context comparisons.Source: Table 2 · p. 6

4.2 Training

• Training uses one epoch of AdamW optimization, LoRA rank 8, and maximum sequence length 8192 for long-context multimodal instruction data.The models are trained on a single 8-A100-40G node for 30 hours and 12 hours, respectively.Source: §4.2 · p. 6

4.3 Evaluation

• Evaluation uses accuracy for multiple-choice tasks and Rouge-L for text-generation tasks across six long-context multimodal tasks.Standard multimodal benchmarks use their official metrics and evaluation splits.Source: §4.3 · p. 6

5.1 Long Image and Text Context

• SEEKER significantly surpasses larger open-source MLLMs across all four long-form multi-image input tasks and, on average, outperforms GPT-4V.SEEKER-TINY ranks second best.Source: §5.1 · p. 6
• SEEKER achieves the best performance on long-context tasks requiring long-form text output, with its models gaining 38.1% from small-scale verbatim-task instruction data.The comparison discusses models’ differing strengths on single-image and document-reading tasks.Source: §5.1 · pp. 6–7
• When text exceeds 4,000 tokens but image tokens remain below 4,000, the VLM outperforms the LLM by 4 to 8 percentage points.VLM performance declines once image tokens exceed 4,000, but remains slightly superior to the LLM.Source: §5.1 · p. 7, Table 3 · p. 7

5.2 General Multimodal Understanding Benchmark

• SEEKER maintains on-par performance with similarly sized models on short-context multi-image tasks and other MLLMs on general single-image benchmarks.This preservation occurs despite excluding general single-image instruction data from continual instruction tuning.Source: §5.2 · pp. 7–8

6.2 Inference Efficiency

• SEEKER is approximately three times faster than the OCR-based approach in the longest context scenario.The speedup comes from eliminating the time-consuming OCR step before reasoning and generation.Source: §6.2 · p. 8, Figure 5 · p. 8

6.3 Qualitative Showcases

• Figure 4 compares image-token and OCR-text token-count distributions, showing that image tokens are more compact within an 8192-token context.The density plot uses blue for image tokens and orange for OCR text.Source: Figure 4 · p. 8
• SEEKER demonstrates coherent long-context text generation, sentence retrieval, grounding across multiple images, and multi-frame video understanding.The showcases compare SEEKER-7B with DeepSeek-VL-7B for document-title identification.Source: §6.3 · p. 8, Figure 6 · p. 9

7 Conclusion

• SEEKER advances long-context multimodal comprehension by processing lengthy texts in visual formats with continual instruction-tuning on extended-context tasks.The conclusion frames this as an advance over existing multimodal large language models.Source: §7 · p. 9
• SEEKER combines better long-context extrapolation and inference efficiency than an OCR-based approach while generalizing to domains including video question answering.The paper presents this as a broad conclusion about its long-context capabilities.Source: §7 · p. 9

Appendix

• Figure 7 reports the training loss curve.No further interpretation is supplied in the passage.Source: Figure 7 · p. 14

A.1 Training Loss Curve

• SEEKER stabilizes at a lower training loss than SEEKER-TINY after a smoother and more consistent decrease.Both models show a rapid initial loss drop, but SEEKER’s curve remains smoother.Source: §A.1 · p. 14

A.2 Evaluation Benchmarks and Metrics

• The evaluation covers four long-form multi-image input tasks: Index, SentRetrie, ArxivQA, and PassKey.These tasks test image indexing, sentence retrieval, document question answering, and masked-word retrieval.Source: §A.2 · p. 14
• Two long-form text-output tasks, ArxivVerb and WikiVerb, require verbatim extraction from visual document content.Source: §A.2 · p. 14

B.1 Tradeoff of Compact Context Length and High Resolution

• High-resolution GPT-4-Vision uses more image tokens and trades away long-document extrapolation performance on First-Sentence-Retrieval.The low-resolution setting preserves performance better, while high resolution can decline faster than OCR-extracted text input.Source: §B.1 · p. 15, Figure 8 · p. 15
• Figure 8 visualizes performance on the First-Sentence-Retrieval task.Source: Figure 8 · p. 15
• Figure 9 presents the Index task visualization.Source: Figure 9 · p. 16

C.1 Task Examples

• The section visualizes full examples for the long-form multi-image input and long-form text-output tasks.Source: §C.1 · p. 16
• The task examples include PassKey, ArxivQA, and ArxivVerbatim visualizations.Source: Figure 10 · p. 17, Figure 11 · p. 18, Figure 12 · p. 19
• Additional examples cover WikiVerbatim and SentRetrie.Source: Figure 13 · p. 20, Figure 14 · p. 21

D.1 Limitations

• Compressing textual information into visual tokens may overlook precise textual understanding.The paper suggests hybrid encoding strategies to balance token compression with preservation of essential information.Source: §D.1 · p. 22

D.2 Societal Impact

• SEEKER may perpetuate training-data biases, while unequal access could exacerbate information disparities.The paper calls for further bias research and broader affordability and availability.Source: §D.1 · p. 22, §D.2 · p. 22