Source-linked AI summary
When History Is Multimodal: Rethinking Context Management for Long-Horizon Agents
Jiaqi Su, Cong Pang, Jiawei Hong, Tiankuo Yao, Zixuan Chen, Xin Lou, Lewei Lu
TL;DR
Long-horizon agents need bounded context representations, while prior optical-memory work leaves unclear whether visual rendering is competitive as a fixed manager and whether native multimodal evidence should be preserved. The paper formulates context management as a budget-constrained history transformation, evaluates Visual Rendering fairly, and proposes training-free VERA. VERA matches existing managers on text-centric tasks and achieves the highest accuracy on multimodal tasks while substantially reducing non-cache tokens.
Problem
Prior optical-memory work mainly renders textualized histories, leaving unclear how untrained Visual Rendering compares with fixed managers and whether native multimodal evidence adds value.
Method
The paper defines context management as a budget-constrained history transformation and implements VERA as deterministic, training-free rendering that preserves native visual observations and structured artifacts.
Results
Across nearly all benchmarks, VERA reduces cumulative non-cache tokens versus No Compression, matches existing managers on text-centric tasks, and achieves the highest accuracy on multimodal tasks.
Takeaways & Limitations
The results support preserving domain-specific multimodal structures, rather than treating visual memory only as a carrier for textualized history.
Takeaways & Limitations
VERA’s multimodal benefit is not established for audio, video, raw database state, or other modalities, and rendering every modality into pixels may not preserve task-relevant structure.
Abstract
from arXiv · showhide
Long-horizon agents need a context manager to compress growing interaction histories into a bounded working context, via passive strategies or active strategies that decide how memory is accessed and reorganized. Meanwhile, prior optical-memory work mainly treats pixels as a dense codec for textualized histories, often presupposing that rendering context into optical memory incurs a significant performance drop relative to text, thus coupling this representation with SFT, self-distillation, or reinforcement learning to close this gap, leaving unresolved (i) how visual rendering performs as a context manager under a fair, controlled comparison, and (ii) whether this carrier offers a native advantage when history is inherently multimodal. In this paper, we formulate context management as a budget-constrained history transformation and introduce Visual Rendering (VR) as a representational context manager. Under a shared harness, policy model, trigger, and task domain, we evaluate VR on four text-centric and three multimodal benchmarks against four baselines (No Compression, Discard-All, Sliding Window, Summarization), finding visual memory is a natural carrier of native visual evidence. Building on this finding, we propose VERA (Visual Evidence-Retaining strategy for long-horizon Agents), a training-free context manager built on deterministic rendering with no exposed memory operations: on text-centric benchmarks it renders textual history as VR does, while on multimodal benchmarks it retains native visual observations instead of translating them into text. Across nearly all benchmarks, VERA cuts cumulative non-cache tokens by 31.5%-63.1% versus No Compression, matches existing managers on text-centric tasks, and achieves the highest accuracy among all baselines on multimodal tasks, supporting a modality-preserving view of long-horizon context management.
1 Introduction
Long-horizon agents need bounded representations of growing interaction histories, but prior optical-memory work leaves unclear how untrained visual rendering compares fairly with fixed managers and whether native multimodal evidence provides an advantage. The paper defines a controlled comparison and introduces VERA to preserve domain-native evidence during compression.
- Long-horizon trajectories can exceed usable context windows, increase inference cost, and bury relevant evidence in long prompts.This motivates explicit conversion of growing histories into bounded working contexts.
- Prior optical-memory work mainly renders textualized history and leaves untrained Visual Rendering’s comparative performance and native multimodal value unclear.The unresolved comparison concerns both standard fixed managers and inherently visual evidence.
- Context management is formulated as a budget-constrained history transformation that separates retained content from its representation and access.Visual Rendering changes representation while holding the trigger, recent context, and downstream policy fixed.
- VERA is a training-free, domain-aware manager that preserves native images and structured analytical artifacts alongside rendered text.It uses deterministic rendering without a learned rendering policy, separately trained reader, or agent-exposed memory operation.
- The controlled comparison evaluates Visual Rendering against No Compression, Discard-All, Sliding Window, and Summarization under a shared policy and agent protocol.The study includes four text-centric benchmarks and examines token efficiency and competitive performance.
2 Related Work
Related work covers long-horizon agents, fixed and active context managers, and optical-context systems that encode textualized history as pixels. This paper distinguishes those approaches from modality-preserving retention of native visual and structured evidence.
- Long-horizon agent systems support extended tool-use loops, while multimodal benchmarks require reasoning over retrieved images, crops, tables, and charts.Flattening tables and charts into prose can weaken their structure.
- Context managers differ in what they retain, how they encode it, how they update memory, and how memory is accessed.Fixed managers use predetermined transformations, whereas other systems condition transformation or access on current state.
- Sliding Window evicts older tool responses, Summarization produces textual abstractions, and Visual Rendering encodes managed history as pixels.No Compression and Discard-All serve as full-retention and reset references.
- Much optical-context work treats visual representation as an OCR-style codec for textualized history and optimizes rendering or access.The related systems focus on visual-token density, layout, compression, selection, or learned policies.
3 Method
The method defines context management as producing a budget-respecting representation of interaction history, then implements Visual Rendering and VERA to preserve native visual evidence. VERA renders history deterministically while retaining recent interactions and modality-specific artifacts.
- Context management: A context manager transforms pre-action history H_t into working context under budget B.The resulting context must respect the combined textual and visual input-cost constraint.
- Context management: A genuine manager must produce a non-empty representation that is strictly cheaper than the raw history.This criterion excludes trivial pass-through and reset transformations.
- Manager types: No Compression and Discard-All are boundary controls, while Sliding Window, Summarization, and Visual Rendering are fixed managers.The fixed managers apply predetermined rules without agent-issued edits.
- Manager types: Active managers can dynamically select, drop, rewrite, or summarize history spans at each trigger.Persistent-memory designs may additionally route evicted content into an external store that can be queried or modified later.
- Native visual evidence: Native visual evidence is decision-relevant structure carried by pixels or spatial arrangement without first generating prose.Examples include images, crops, zoomed views, rendered tables, charts, and document pages.
- VERA implementation: VERA uses deterministic chronological rendering, keeps recent interactions in original form, and places native visual observations beside associated interactions.It uses the policy model’s existing vision encoder and preserves provenance through role and tool labels.
4 Experiments
The experiments compare fixed context managers under shared infrastructure across text-centric and multimodal benchmarks. Visual Rendering is highly token-efficient and competitive on textual histories, while VERA’s preservation of native visual evidence yields the strongest multimodal results; older-history fidelity has a non-monotonic accuracy-cost trade-off.
- Experimental setup: The evaluation uses a shared harness, policy model, trigger, and task protocol to compare VERA with No Compression, Discard-All, Sliding Window, and Summarization across text-centric and multimodal tasks.Metrics include task success rate and cumulative non-cache token consumption.
- RQ1: Text-centric histories: 44.7%, 54.8%, 52.1%, and 31.5% are Visual Rendering’s token reductions relative to No Compression across the four text-centric benchmarks.On BrowseComp-EN, accuracy improves by 16.0 points while tokens fall from 1.03M to 0.57M.
- RQ1: Text-centric histories: Plain Visual Rendering achieves the highest aggregate text-centric accuracy at 74.2% and uses the fewest cumulative non-cache tokens on all four benchmarks.It remains training-free and is not uniformly best on every individual benchmark.
- RQ2: Native visual evidence: VERA attains 31.7% on BrowseComp-V3 versus 27.3% for No Compression, while reducing cumulative tokens from 2.61M to 1.22M.On MM-BrowseComp, it improves accuracy from 23.3% to 28.5% while reducing tokens from 3.74M to 1.38M.
- RQ2: Native visual evidence: Restoring native images, screenshots, localized crops, and analytical artifacts raises accuracy by 6.7 points on BrowseComp-V3 and 7.0 points on MM-BrowseComp.On AgenticDataBench, direct artifact rendering raises the official average score from 0.37 to 0.41 while reducing tokens from 419K to 406K.
- RQ3: Retention and fidelity: Intermediate rendering scales reach the highest observed accuracy, with 0.7× and 0.8× at 71.0%, while 0.8× has the lowest cumulative tokens at 1.03M and the fewest API calls at 56.5.Older-history retention raises accuracy from 62.9% to 64.0% and reduces cumulative tokens by 20.4% relative to recent context alone; the main experiments use 0.75×.
5 Conclusion
The paper frames context management as a budget-constrained transformation and argues that domain-specific multimodal structures should remain available to long-horizon agents.
- Context management constructs a nonempty representation of managed history under a budget, excluding full retention and full removal as boundary controls.
- Plain Visual Rendering is a strong, training-free default on text-centric benchmarks, while VERA preserves native visual observations and structured analytical artifacts.
- The paper concludes that future context managers should combine domain-aware representations with learned selection, pruning, and access policies.
Limitations
The limitations concern extreme-horizon scalability and the restricted multimodal scope of VERA, leaving broader modality support and learned selection open.
- Scalability at Extreme Horizons: At sufficiently long horizons, VERA may retain dozens of rendered observations because it uses bounded-window retention rather than recursive compression.Its compression ratio is therefore not guaranteed to match recursive summarization as horizon length grows.
- Visual Scope and Multimodal Dependence: VERA requires a vision-language policy model, and the multimodal experiments do not establish benefits for audio, video, raw database state, or other modalities.
- Visual Scope and Multimodal Dependence: Rendering every modality into pixels may not preserve task-relevant structure, motivating latent-state alternatives whose long-horizon evidence retention remains open.
- Discard-All can improve performance through repeated locally conditioned attempts, but it sacrifices evidence that must be integrated across trajectory segments.
B.1 Controlled Evaluation Protocol
The evaluation holds the agent protocol and policy configuration fixed while varying how managed history is treated, including deterministic visual rendering.
- All strategies use the same harness, policy backbone, task prompt, tool interface, stopping rule, retained context, and trigger configuration.The policy context window is 128k tokens, with activation at 64k active-context tokens.
- No Compression retains all history, Discard-All removes managed history, Sliding Window evicts older tool responses, and Summarization recursively replaces history with a textual summary.
- VERA deterministically renders chronological managed history while retaining recent context in native form and keeping available screenshots, tables, charts, and analytical outputs visual.Each model call contains at most 10 rendered images, and the main experiments use a 0.75× canvas scale.
B.2 Evaluation Metric and Timeout Handling
The evaluation defines benchmark accuracy as a question-count-weighted average and reports token usage across all evaluated trajectories. Timeout rollouts are retried rather than discarded, ensuring identical treatment across strategies.
- Evaluation Metric: Benchmark-level accuracy is computed from R = 3 independent rollouts per question.The evaluation uses avg@3 for robustness.
- Evaluation Metric: Combined accuracy is the question-count-weighted average across benchmarks.The text-centric evaluation totals 700 questions across the reported benchmark counts.
- Timeout Handling: Timeout rollouts resume from existing context and are retried up to 5 times before scoring, rather than being discarded.The protocol is applied identically across treatments to avoid optimistic bias toward easier timeout-free cases.
- Token Accounting: Cumulative non-cache tokens include uncached input and output tokens across every model call in each trajectory.Cache-read tokens are excluded, and input and output tokens are added rather than subtracted.
- Token Accounting: A compression-aware diagnostic measures effective context across compression segments and is not interchangeable with the main-table non-cache metric.This diagnostic analyzes repeated compression rather than API tokens processed or billed.
C Extended Analysis of Older Visual Context and Fidelity
The extended analysis tests whether retaining older visual history helps and how rendering fidelity affects performance and cost. Moderate downscaling performs best in the reported BrowseComp study, motivating a 0.75× scale for the main experiments.
- Rendering Fidelity: Full resolution contributes approximately 77.8% more visual tokens than a 0.75× canvas under the patchtoken approximation.The approximation is visual-only and may differ from provider accounting because of adaptive tiling and preprocessing boundaries.
- Older History: Older visual history improves accuracy by 1.1 points while lowering token, API-call, and turn costs.The matched comparison contrasts the 0× recent-context-only condition with retained older history.
- Rendering Fidelity: 0.8× uses 1.03M cumulative tokens and 56.5 average API calls in the reported scale comparison.These are the lowest reported cost figures among the retained scale settings described.
D Additional Turn-Budget Efficiency Analysis
The turn-budget analysis compares cumulative task success for Visual Rendering and other strategies on BrowseComp. Visual Rendering accumulates successful completions faster and maintains the highest curve over longer trajectories.
- Turn-Budget Efficiency: Visual Rendering accumulates successful completions more rapidly than the comparison strategies as turn budget increases.Figure 3 plots cumulative task success rate against available turn budget.
- Turn-Budget Efficiency: Visual Rendering sustains the highest cumulative task-success curve across longer trajectories.The analysis uses BrowseComp and compares Visual Rendering with the listed comparison strategies.