PaperlayerView exact revision on arXiv

Source-linked AI summary

MemOCR: Layout-Aware Visual Memory for Efficient Long-Horizon Reasoning

Yaorui Shi, Shugui Liu, Yu Yang, Wenyu Mao, Yuxin Chen, Qi GU, Hui Su, Xunliang Cai, Xiang Wang, An Zhang

arXiv:2601.21468v5cs.AI

TL;DR

Long-horizon agents must compress growing histories into finite context windows, but textual memory couples information content to uniform token cost. MemOCR uses rich-text drafting, visual rendering, and budget-aware reinforcement learning to allocate information density spatially; across QA benchmarks, it improves robustness and effective context utilization under tight budgets.

  • Problem

    Textual memory scales linearly with history and assigns uniform token cost, limiting selective retention of crucial evidence under finite context budgets.

  • Method

    MemOCR incrementally drafts structured rich-text memory, renders it into a 2D image, and trains drafting and reading with budget-aware reinforcement learning.

  • Results

    Across multi-hop and single-hop QA, MemOCR outperforms text-memory agents and has substantially smaller performance drops as budgets tighten, yielding roughly an 8× improvement in effective context utilization.

  • Takeaways & Limitations

    Adaptive visual layout makes memory compression more effective under extreme budgets while preserving access to crucial evidence.

  • Takeaways & Limitations

    The authors identify privacy and security risks, high-stakes-domain safety concerns, and possible hallucinations from unreliable visual rendering or OCR-style reading.

Abstract

from arXiv · show

Long-horizon agentic reasoning necessitates effectively compressing growing interaction histories into a limited context window. Most existing memory systems serialize history as text, where token-level cost is uniform and scales linearly with length, often spending scarce budget on low-value details. To this end, we introduce MemOCR, a multimodal memory agent that improves long-horizon reasoning under tight context budgets by allocating memory space with adaptive information density through visual layout. Concretely, MemOCR maintains a structured rich-text memory (e.g., headings, highlights) and renders it into an image that the agent consults for memory access, visually prioritizing crucial evidence while aggressively compressing auxiliary details. To ensure robustness across varying memory budgets, we train MemOCR with reinforcement learning under budget-aware objectives that expose the agent to diverse compression levels. Across long-context multi-hop and single-hop question-answering benchmarks, MemOCR outperforms strong text-based baselines and achieves more effective context utilization under extreme budgets.

1. Introduction

MemOCR replaces linear textual memory with layout-aware visual memory, allocating adaptive information density so crucial evidence remains readable under tight budgets. Budget-aware reinforcement learning improves robustness across long-context QA settings.

  • Textual memory imposes uniform information density, forcing auxiliary details and crucial evidence to consume comparable token space.
  • MemOCR represents history as a 2D image and allocates visual space non-uniformly through typography and layout.Prominent formatting preserves crucial evidence while smaller text compresses auxiliary details; resolution controls the budget-fidelity tradeoff.
  • The agent incrementally updates rich-text memory, then renders it into an image used as its sole working context for answering queries.
  • MemOCR outperforms text-memory agents when budgets suffice and suffers substantially smaller performance drops as budgets tighten.The reported gains span multi-hop and single-hop QA across varying context lengths and memory budgets.
  • Visual salience is functionally important: weakening or removing layout-based emphasis harms low-budget robustness.MemOCR also learns to place more important information in more visually accessible regions.

2. Preliminaries

Long-horizon agents must answer from sequential histories that grow beyond finite attention windows. Textual summaries reduce redundancy but retain a uniform token cost that limits non-uniform budget allocation.

  • 2.1. Problem Formulation: The agent receives sequential text chunks and must produce an accurate answer from cumulative history under a finite working context.
  • 2.1. Problem Formulation: As interaction history grows, raw-history conditioning eventually exceeds the underlying LLM’s effective attention window.
  • 2.1. Problem Formulation: Textual summary memory iteratively refines a query-conditioned state and generates the final answer from the latest memory.
  • 2.2. Memory Budget and Uniform Information Density: A memory budget limits the working context available at inference time.
  • 2.2. Memory Budget and Uniform Information Density: Text-based memory assigns every token the same budget cost, so auxiliary details compete directly with crucial evidence.

3. Method: MemOCR

MemOCR drafts a structured rich-text memory, renders it as a visual memory image, and reads that image under a controllable visual-token budget. Budget-aware QA objectives train layouts to preserve crucial evidence while retaining detail when resources permit.

  • MemOCR uses a two-stage lifecycle: text-domain memory drafting followed by vision-domain memory reading after rendering.
  • Memory Drafting: The drafter incrementally maintains persistent rich text whose structure and formatting encode visual priority.Headings, indentation, bolding, and font size determine how memory components compete for canvas space.
  • Memory Drafting: Drafting is budget-agnostic, producing one rich-text memory whose internal salience structure enables non-uniform allocation after rendering.
  • Memory Reading: A lightweight renderer transforms the rich-text memory into a 2D image, whose layout and typography control visual-token cost.For text length L and font scale s, occupied area scales as O(L · s^2).
  • Memory Reading: Resolution manipulation controls the effective budget by limiting the rendered image’s visual patch tokens.
  • Budget-Aware Training Objectives: Budget-aware reinforcement learning combines standard QA with compressed-memory and detail-oriented objectives.Augmented Memory preserves crucial evidence under severe compression, while Augmented Question retains access to auxiliary details when tokens suffice.
  • Budget-Aware Training Objectives: Reading uses separate task-specific advantages, whereas drafting uses an aggregated advantage to learn one layout serving all scenarios.

4. Experiments

Across long-context QA and conversational-memory evaluations, MemOCR maintains stronger performance and robustness than text-based methods, especially under severe memory budgets. Ablations indicate that visual layout and budget-aware training drive this behavior.

  • Overall Performance (RQ1): MemOCR attains the highest average accuracy from 10K to 100K contexts, reaching 74.6% versus 67.8% for the strongest baseline at 10K with full budget.The paper also reports poor HotpotQA performance at 30K and 100K contexts.
  • Overall Performance (RQ1): At 16 tokens, MemOCR preserves 62.2% average accuracy with only a 16.6% relative drop, while MemAgent falls from 67.8% to 31.6% as its budget tightens.This comparison uses 10K-context evaluations and shows substantially greater low-budget robustness for MemOCR.
  • Overall Performance (RQ1): MemOCR remains superior or comparable when textual baselines receive 4× more tokens, achieving 62.2% at B = 16 versus MemAgent’s B = 64 result by 11.5 points.This tests whether gains persist under token-budget fairness rather than arising solely from a larger vision-side budget.
  • Fairness and Generalization: Replacing naive truncation with summarization raises MemAgent from 31.6 to 58.5 on B = 16, yet MemOCR still leads across all budgets.The comparison indicates that MemOCR’s advantage persists against an optimized text-compression baseline.
  • Visual Robustness and Mechanism: MemOCR’s layout is functionally important: removing visual layout causes additional low-budget degradation, while layout-guided regions preserve crucial evidence under downsampling.Prominent regions outperform detailed regions under compression, and RL shifts denser evidence into the crucial region by approximately 1.8×.
  • Visual Robustness and Mechanism: At 8 tokens, MemOCR achieves comparable accuracy to baselines at 64 tokens, corresponding to an 8× reduction in memory tokens.The result is reported as an extreme-budget token-efficiency comparison.
  • Ablation Study over Training Objectives: Explicit training is necessary for budget-robust layouts, and combining query- and memory-augmentation objectives produces larger low-budget gains than standard training alone.Without training signals, the desired layout arrangement does not emerge reliably.

5. Conclusion and Future Work

MemOCR reframes agent memory as a spatial 2D canvas that decouples information density from token cost, supporting robustness under extreme budgets. Future work targets broader agent settings, lifelong updates, and more flexible rich-text formats.

  • Conclusion: MemOCR shifts agentic memory management from linear textual streams to a spatial 2D canvas that dynamically manipulates visual layout and resolution.The paper terms this representation visual memory.
  • Future Work: Future work will generalize visual memory beyond QA to planning and tool-augmented reasoning and study long-term stability under lifelong updates.The authors also plan more flexible rich-text formats such as HTML for budget allocation.

Impact Statement

MemOCR advances long-horizon agentic reasoning through visual memory and adaptive context allocation, while stronger memory can amplify privacy, security, safety, and OCR-related risks.

  • Impact Statement: MemOCR represents interaction history on a 2D canvas and allocates limited context budgets through adaptive visual compression.The approach is intended to improve effective context utilization and may enable more long-horizon multimodal agents.
  • Impact Statement: Stronger long-horizon memory can amplify privacy and security concerns when sensitive information is stored, rendered, or exposed through outputs or logs.The paper also identifies risks for high-stakes domains where safety solutions are designed for text-only interactions.
  • Impact Statement: Visual rendering and OCR-style reading may introduce failure modes that produce hallucinated responses when key evidence is not reliably recovered.The paper notes that these risks can be amplified by improved memory capacity rather than being unique to MemOCR.
  • Impact Statement: Proposed mitigations include strict retention policies, user consent, transparency, access control, encryption, and robustness and bias evaluation across demographics and domains.

B. Limitations

MemOCR uses a two-stage rich-text drafting and visual reading pipeline with budget-aware reinforcement learning, but its vision/OCR robustness, transferability, and deployment overhead remain constrained.

  • Limitations: MemOCR’s limitations include dependence on vision/OCR robustness, task-specific layout policies, and additional computational overheads.The appendix supplies engineering details for the drafting, rendering, budget-control, reading, training, and evaluation pipeline.
  • Pipeline: The agent drafts persistent rich-text memory by updating content and encoding priority through headings, bullets, bolding, and other formatting.Drafting is budget-agnostic and produces a semantically complete memory before rendering.
  • Pipeline: A deterministic renderer converts the final rich-text memory into a 2D image, then downsamples it so visual tokens remain within budget B.The renderer uses Markdown, HTML, fixed CSS, and a headless Chromium screenshot process.
  • Pipeline: At reading time, the agent receives only the budgeted memory image and question, so task-relevant information must be recovered from the image.No raw history or original long context is provided during this stage.
  • Training: MemOCR uses Qwen2.5-VL-7B-Instruct with full-parameter updates in BFloat16 and FSDP, trained on chunked long-context QA data.The training setup uses HotpotQA-derived contexts and updates persistent memory after each chunk.
  • Training: Budget-aware GRPO training combines moderate-budget QA, aggressively compressed QA, and high-resolution detail-oriented QA objectives.The objectives train the layout and salience policy across different compression regimes.

C.3. Evaluation Details

Evaluation compares MemOCR with reproduced textual baselines under matched long-context construction, decoding, and memory-budget protocols, with repeated runs and statistical testing.

  • Evaluation Setup: Evaluation uses approximately 10K, 30K, and 100K-token contexts formed from instance documents and sampled distractors.Distractor seeds are fixed per split, and four benchmarks are randomly downsampled to 128 instances because long-context QA is time-consuming.
  • Evaluation Setup: Accuracy is measured with sub-word exact match and reported as means over three independent random-seed runs.Both baselines and MemOCR use stochastic decoding unless otherwise noted.
  • Budget Control: Text baselines are constrained by truncating memory to the first B tokens, while MemOCR resizes its image to at most B visual tokens.This aligns effective context capacity across text and vision modalities.
  • Baselines: The study reproduces MemAgent, Memα, and Mem0 using official releases or documentation and matches their answer model, data construction, splits, and decoding settings.
  • Statistical Verification: Statistical analysis reports means and standard deviations and applies an independent two-sample t-test to averaged accuracy.The resulting p-values and performance gains are summarized in Table 9.
  • Statistical Verification: At B ∈{16, 64}, MemOCR shows significant gains, including +30.6 at 10K/16 tokens, while at B = 1024 and 100K context the gain is +0.9 and not significant (p = 0.3419).The reported gains are largest under severe memory constraints and diminish as memory and context become ample.
  • Complexity: MemOCR does not incur significant computation overhead compared with textual memory in the reported long-horizon complexity analysis.

D.2.1. THEORETICAL ANALYSIS

The theoretical analysis models chunked long-context processing with fixed chunk and memory budgets, finding linear drafting growth in N and matching N-scaling for textual and visual memory.

  • Notation and Assumptions: The full context contains N tokens split into T chunks of approximately L ≈N/T tokens, with fixed memory budget B and full self-attention cost O(x^2).Question length is ignored because it is typically small compared with memory.
  • Drafting Complexity: Memory drafting consumes each chunk together with bounded prior memory, represented by the update M_t ∼π(· | M_t−1, C_t).The memory budget B bounds the number of in-context memory tokens shown to the updater.
  • Drafting Complexity: With fixed L and B, memory drafting scales linearly with long-context length N.
  • Reading Complexity: Textual-memory reading feeds a memory of length B into the language model, while visual reading processes an image producing at most B visual tokens.Visual answering includes vision-head processing and language-model attention over the B visual tokens.
  • Overall Complexity: MemOCR and textual memory have the same theoretical complexity scaling in N through T ≈N/L.The overall comparison combines drafting and reading terms for both modalities.

D.2.2. EMPIRICAL ANALYSIS

MemOCR’s visual-memory rendering adds lightweight overhead and remains runtime-competitive with a textual-memory baseline across long-context settings.

  • MemOCR and MemAgent show near-linear runtime growth with context length under the long-context protocol.MemOCR remains comparable to, and is faster than the text baseline in some cases, at long contexts.
  • 1 second per 68 samples and 0.175 extra latency characterize the lightweight overhead of MemOCR’s deterministic image-rendering process.Rendering is linear in output canvas size and does not invoke an LLM.

D.3. Additional Ablation Studies

Additional ablations show that budget-aware RL, multi-hop training data, layout-aware memory, and visual formatting support MemOCR’s robustness and transfer across settings.

  • Backbone scaling: 7B+RL matches or exceeds 32B/72B non-RL backbones in most long-context settings, indicating that budget-aware RL is more effective than naive model scaling.The advantage is especially pronounced at 100K context on multi-hop benchmarks.
  • Budget robustness: MemOCR shows much smaller relative performance drops than non-RL baselines as the memory budget shrinks, remaining reliable at the extreme 16-token budget.
  • Cross-model transfer: 87.7 accuracy at 10K shows GPT-4o with MemOCR memory surpassing Qwen2.5-VL’s native-reader score of 84.8 on HotpotQA.The layout-aware variant consistently outperforms the variant without layout.
  • Aspect-ratio robustness: Less than 2 points of difference across aspect-ratio configurations suggests readers rely primarily on typographic cues rather than absolute spatial coordinates.
  • General language understanding: A 1.4-point MMLU drop relative to the base Qwen2.5-VL-7B-Instruct model indicates limited degradation in broader language understanding after RL training.
  • Training-data composition: HotpotQA training improves both benchmarks, whereas NQ training yields 66.1 versus 45.3 on NQ at B=1024 but degrades HotpotQA to 53.9 versus 67.2.The authors interpret this as evidence that multi-hop data transfers better, while single-hop training overfits to a narrow distribution.
  • Training dynamics: MemOCR overtakes MemAgent after approximately 600 GRPO steps and maintains a growing reward advantage through convergence.MemAgent converges slightly faster during the first 400 steps.

E. Bad Case Analysis

MemOCR’s bad cases arise when low-budget rendering makes fine-grained comparison details unreadable or when excessively long memories force text below visual resolution.

  • The analysis identifies two representative failure modes despite MemOCR’s strong performance in emphasizing crucial information through layout generation.
  • Failure Mode A: Fine-grained detail loss: Comparative questions can fail when entity headers remain legible but crucial attribute text collapses into unreadable pixel noise under low-budget downsampling.The layout prioritizes entity names over descriptions, causing incorrect answers for detailed attribute comparisons.
  • Failure Mode B: Memory capacity overflow: Excessively long rich-text memories can trigger font scaling below the visual encoder’s readability threshold, causing information loss.The cited example involves a repeated-word generation issue and a history exceeding 2000 characters.
Loading 2601.21468v5…