PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentOCR: Reimagining Agent History via Optical Self-Compression

Lang Feng, Fuchao Yang, Feng Chen, Xin Cheng, Haiyang Xu, Zhenglin Wan, Ming Yan, Bo An

arXiv:2601.04786v3cs.LGcs.AI

TL;DR

Long-horizon LLM agents accumulate textual histories that inflate token and memory costs. AgentOCR renders those histories as compact visual representations, adds segment optical caching and RL-trained self-compression, and preserves most task performance while substantially reducing token consumption.

  • Problem

    Growing multi-turn textual histories create token, memory, latency, and compute burdens for LLM agents.

  • Method

    AgentOCR renders observation-action histories as visual memory, reuses hashable segments through optical caching, and learns compression factors with compression-aware RL rewards.

  • Results

    AgentOCR preserves over 95% of text-based agent performance while reducing token consumption by over 50% across ALFWorld and search-based QA.

  • Takeaways & Limitations

    Visual history can retain most decision-making capability while reducing token consumption, providing a resource-efficient alternative to text-only processing.

  • Takeaways & Limitations

    The paper identifies limitations for AgentOCR but does not specify their concrete scope in the supplied passages.

Abstract

from arXiv · show

Recent advances in large language models (LLMs) enable agentic systems trained with reinforcement learning (RL) over multi-turn interaction, but practical deployment is bottlenecked by rapidly growing textual histories that inflate token and memory costs. We introduce AgentOCR, a framework that exploits visual tokens' superior information density by representing the accumulated observation-action history as a compact rendered image. To make multi-turn rollouts scalable, AgentOCR proposes segment optical caching. By decomposing history into hashable segments and maintaining a visual cache, this mechanism eliminates redundant re-rendering. Beyond fixed rendering, AgentOCR introduces agentic self-compression, where the agent actively emits a compression rate and is trained with compression-aware reward to adaptively balance task success and token efficiency. We conduct extensive experiments on challenging agentic benchmarks, ALFWorld and search-based QA. Remarkably, AgentOCR preserves over 95\% of text-based agent performance while substantially reducing token consumption (>50\%), yielding consistent token and memory efficiency. Further analysis validates a 20$\times$ rendering speedup from optical caching and effective self-compression balancing. Our code is available at https://github.com/langfengQ/AgentOCR.

1 Introduction

Agentic RL systems face rapidly expanding interaction histories that increase token, latency, and compute costs. AgentOCR replaces textual history with compact visual representations, caching segments and learning compression to preserve performance while reducing overhead.

  • Motivation: Long-horizon agent interactions accumulate extensive histories, exhausting context budgets and increasing inference latency and compute costs.The burden includes expensive attention prefill and KV-cache management.
  • AgentOCR: AgentOCR represents accumulated observation-action history as a compact image and conditions the agent’s policy on this visual history.
  • AgentOCR: Segment optical caching reuses previously rendered history segments through a hash-based cache, avoiding redundant processing as rollouts expand.
  • AgentOCR: Agentic self-compression lets the agent select a compression factor through RL to balance task success and token cost.
  • Results: Over 50% lower token consumption accompanies preservation of over 95% of text-agent task performance across ALFWorld and search-based QA.Peak token reduction reaches 80%, while optical caching accelerates rendering by over 20×.

2 Related Work

Prior work spans reinforcement learning for agent behavior, OCR and vision-text compression, and methods for efficient long-context processing. These lines of research motivate AgentOCR’s focus on compressing persistent agent histories.

  • Reinforcement Learning: RL methods such as PPO, GRPO, and related variants are used to align LLM agents and improve behavior in complex environments.
  • OCR: OCR has evolved from document and image text extraction toward end-to-end image-to-text modeling for complex scenarios.
  • Vision-Text Compression: Recent OCR-based approaches explore vision-text compression for extremely long contexts, but this research direction remains in its early stages.
  • Long-Context Modeling: Long-context research addresses persistent agent histories through sparse or hierarchical attention, recency-biased positions, prompt compression, and retrieval-based methods.

3 Preliminaries

The paper models an LLM agent as a stochastic policy interacting with an environment over a finite horizon, with actions conditioned on task instructions and accumulated history. RL training uses group-relative advantages and a clipped surrogate objective, but growing histories make optimization increasingly expensive.

  • Problem Setup: An agent interacts with an environment over finite horizon T, receiving observations and producing actions under stochastic policy πθ.
  • Problem Setup: The interaction history is the sequence h_t = (o_1, a_1, o_2, a_2, ..., o_t), which conditions subsequent textual actions.
  • Problem Setup: Actions may include intermediate reasoning or tool invocations, after which the environment returns a reward and the next observation.
  • Agentic Reinforcement Learning: GRPO estimates advantages by normalizing rewards within groups and optimizes a clipped surrogate objective for training stability.
  • Agentic Reinforcement Learning: More than 10k tokens can accumulate in multi-turn search histories, making gradient computation and memory increasingly expensive.

4 AgentOCR

AgentOCR addresses the bottleneck of growing multi-turn histories by rendering interaction history as optical memory, then adds segment caching and agent-controlled compression for scalable, token-efficient rollouts.

  • 4.1 Optical Memory Encoding: AgentOCR renders accumulated interaction history as a compact image and conditions a vision-language policy on that optical history.The renderer maps textual history to an RGB image used alongside the task input.
  • 4.2 Segment Optical Caching: Segment optical caching splits history into segments, stores rendered segments under content keys, and assembles optical memory by stacking their images.Previously seen segments are reused, while only cache misses require rendering.
  • 4.2 Segment Optical Caching: Cache hits require lookup and image stacking, while per-step rendering costs O(U_t) segment renders instead of repeatedly rendering the full history.U_t denotes cache-miss segments; repeated content can therefore reduce rendering overhead when U_t ≪ K_t.
  • 4.3 Agentic Self-Compression: AgentOCR exposes compression as a policy-selected factor that downscales rendered history images and trades visual information density against token cost.The agent emits a compression factor c_t alongside environment interaction, and larger spatial downsampling reduces visual tokens.
  • 4.3 Agentic Self-Compression: Compression-aware reinforcement learning adds a success-conditioned efficiency reward whose weight controls the trade-off between task performance and compression efficiency.An intermittent reward schedule is used to limit overly greedy compression while preserving pressure toward task completion.

5 Experiment

AgentOCR is evaluated on ALFWorld and search-based QA for performance, token efficiency, compression trade-offs, caching, and self-compression. Across these experiments, it preserves task performance while substantially reducing token and rendering costs.

  • Benchmarks and baselines: AgentOCR is evaluated on ALFWorld and search-based QA, which represent embodied household manipulation and text-dense multi-turn information retrieval.The study compares optical-history and text-based agents across prompting and reinforcement-learning regimes.
  • Main results: 55% and 70% average token reductions are observed on ALFWorld and search tasks, respectively, although inference-only optical histories initially reduce task performance.The comparison is between “Text” and “OCR” without reinforcement learning.
  • Main results: Over 95% of text-based performance is retained while token consumption falls by >50%, reaching 80.9% savings in peak contexts.On ALFWorld, 3B and 7B AgentOCR models achieve 78.2% and 81.2%; on search tasks, the 7B model achieves 40.1% versus 41.9% for Text+GRPO.
  • Vision-text compression analysis: 99.5% and 95.0% of text-based performance are maintained at approximately 55% token savings for ALFWorld and Search, respectively.Increasing savings to approximately 67% at ct = 2.0 causes significant average-performance degradation.
  • Cache analysis: 17.28× speedup is achieved by naive caching, while segment optical caching reaches 168.77 ms average latency and 26.82% peak-memory saving relative to naive cache.Segment reuse produces −1.23 ms/step time growth and reduces storage to 110.80 MB by caching unique content segments.
  • Self-compression analysis: 381.7 average visual tokens and 78.2% success are achieved with intermittent reinforcement at K = 5, versus 458.1 tokens and 78.5% for the uncompressed visual baseline.Dense reinforcement at K = 1 increases the compression factor to 4.91 and reduces success to 45.3%.

6 Conclusions

AgentOCR uses visual tokens as a compact medium for multi-turn agent history, combining segment optical caching with agentic self-compression. Experiments suggest it retains most decision-making capability while reducing token consumption.

  • AgentOCR explores visual tokens as a compact history medium for multi-turn LLM agents.
  • Segment optical caching reduces rendering overhead, while agentic self-compression adaptively balances cost and fidelity.
  • AgentOCR retains the majority of decision-making capabilities while significantly reducing token consumption on ALFWorld and search-based QA.

Limitations

The paper identifies limitations in the evaluated VLMs, rendering configuration, and history modality. These boundaries leave performance sensitivity and broader applicability insufficiently explored.

  • AgentOCR relies on off-the-shelf Qwen2.5-VL models not specifically designed for OCR-intensive tasks.The study does not evaluate broader VLM architectures or different visual tokenization strategies.
  • The deterministic renderer uses fixed font, spacing, color, and resolution hyperparameters without systematic sensitivity analysis.Suboptimal settings may reduce text legibility or distort layout cues, potentially affecting downstream reasoning.
  • The current design assumes agent histories consist primarily of text that can be rendered as text-as-image.Multimodal histories containing screenshots, plots, diagrams, or tables remain outside the demonstrated scope.

Ethical Considerations

The experiments use public benchmarks and synthesized environments without sensitive personal information, so immediate social impact is limited. The authors identify possible longer-term resource-efficiency benefits but report no real-world deployment.

  • All experiments use established public benchmarks and synthesized environments without sensitive personal information or privacy concerns.
  • The work currently lacks real-world deployment, limiting its immediate social impact.
  • The authors suggest reduced token overhead may support energy-efficient Green AI and future deployment on resource-constrained devices.
  • The provided pseudocode covers segment caching, rendering with requested compression, efficiency-aware reward computation, and policy-selected compression parameters.

B.1 Details of Benchmarks

The evaluation uses ALFWorld for embodied long-horizon household tasks and Search-based QA for single- and multi-hop question answering. Agents select actions from observations and interaction history, with rollout procedures tracking visual history and compression calls.

  • ALFWorld: ALFWorld contains 3,827 embodied household tasks spanning six task categories.
  • ALFWorld: In ALFWorld, the agent selects an action from the current observation and interaction history at each interaction step.
  • Rollout procedure: The rollout algorithm samples an action and compression call, executes both, and continues the interaction loop until the episode ends.
  • Search-based QA: Search-based QA includes single-hop datasets NQ, TriviaQA, and PopQA, plus multi-hop datasets including HotpotQA and 2WikiMultiHopQA.

B.2 Details of Training

The training setup specifies benchmark-specific context limits, rollout configurations, and rendering prompts for text-only and optical-history agents. AgentOCR additionally instructs the model to emit a compression ratio.

  • ALFWorld uses 5120-token prompts for text-only agents and 2048-token prompts for optical-history variants, with 512-token responses and up to 50 steps per episode.
  • Search-based QA uses 14000-token prompts for text-only agents and 4096-token prompts for optical-history variants, with 512-token responses and up to four environment steps.
  • The optical memory is produced by a deterministic renderer with benchmark-specific typographic and semantic-color hyperparameters.
  • Prompt templates use Python-style placeholders for dynamic task and observation content and control tags for reasoning, actions, searches, retrieved information, answers, and compression.AgentOCR uniquely adds <compression> tags for specifying the compression ratio.

C Case Study

The case study illustrates AgentOCR’s multi-turn search interaction: search results accumulate in optical memory, compression factors change across steps, and the agent reaches the correct answer while using visual history compression.

  • The case study follows a multi-turn interaction between the agent and search tool, showing progressive optical-memory accumulation and adaptive compression factors.
  • The prompt figures contrast text-agent and AgentOCR templates for ALFWorld and search-based QA.
  • AgentOCR’s search-based QA prompt represents full history in an image and identifies past queries and results with <search> and <information> tags.
  • The agent first searches for the locations of Teide National Park and Garajonay National Park while emitting a compression factor of 1.2.
  • At the next step, the agent identifies Teide National Park in Tenerife and Garajonay National Park on La Gomera, while emitting a compression factor of 1.1.
Loading 2601.04786v3…