Source-linked AI summary
Buried in Textual Debt: Context Pruning with Visual Evidence Preservation for MLLM Agents
Yuchen Huang, Sijia Li, Jun Zhang, Yi R. Fung
TL;DR
Multi-step MLLM agents can accumulate self-generated text that overshadows visual evidence, creating textual debt. SPARE uses summary-conditioned reverse-KL diagnostics and SFT-enhanced summaries to prune redundant reasoning while preserving evidence. Across visual tool-use benchmarks, it achieves the highest average accuracy among pruning methods while removing 37.89–64.58% of reasoning tokens.
Problem
Long trajectories accumulate self-generated reasoning that can dominate context and weaken reliance on visual evidence, while existing methods do not directly identify functionally necessary historical segments.
Method
SPARE replays the same model with original and compact-summary-conditioned contexts, using reverse-KL residual sensitivity to prune covered reasoning and SFT to improve summary coverage.
Results
Across multi-step visual tool-use benchmarks, SPARE achieves the highest average accuracy among pruning methods while removing 37.89–64.58% of reasoning tokens.
Takeaways & Limitations
Reducing textual dominance restores reliance on visual evidence and mitigates over-conditioning on self-generated language.
Takeaways & Limitations
SPARE is designed for multi-step agents with explicit reasoning and tool-use histories, making application to single-turn or unstructured agents less direct.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) are increasingly deployed as multi-step agents, where explicit reasoning supports task decomposition and tool coordination but also accumulates self-generated text. Over long trajectories, this text can dominate the context and suppress visual evidence, creating textual debt. We observe that reasoning becomes redundant once task-relevant visual evidence is grounded, while stale hypotheses can misguide later inference when grounding remains uncertain. Pruning must therefore remove redundant text without discarding visual evidence. We propose SPARE, a Kullback--Leibler (KL)-guided framework for pruning accumulated reasoning in multimodal tool-use agents. SPARE uses a compact task-state summary as privileged diagnostic context. For each candidate segment, it replays the same model under the original and summary-conditioned contexts. Reverse-KL divergence from on-policy self-distillation (OPSD) then tests whether the summary sufficiently covers the segment without disrupting future reasoning. We further fine-tune the summarizer with supervised fine-tuning (SFT), enabling more compact summaries, broader coverage, and more aggressive pruning. Across multi-step visual tool-use benchmarks, SPARE achieves the highest average accuracy among pruning methods while removing 37.89--64.58\% of reasoning tokens. This favorable accuracy--context trade-off shows that reducing textual dominance restores reliance on visual evidence and mitigates over-conditioning on self-generated language.
1 Introduction
Long-horizon MLLM agents accumulate self-generated reasoning that can overshadow visual evidence, creating textual debt. SPARE selectively prunes redundant reasoning while preserving evidence, improving accuracy and visual grounding across benchmarks.
- Accumulated self-generated text can dominate the context, conditioning agents on their own plans and assumptions rather than newly returned visual evidence.
- Reasoning becomes textual debt when stale or weakly grounded language contributes little new evidence while competing with image tokens for attention.
- Existing visual-token compression, concise-reasoning, and generic summarization methods do not identify which historical reasoning segments remain functionally necessary.
- SPARE uses summary-conditioned KL divergence to estimate functional redundancy and selectively forget reasoning whose information is already consolidated.
- SFT produces more compact task-state summaries, enabling broader coverage and more aggressive pruning while preserving task-relevant information.
- SPARE improves accuracy and restores visual-evidence influence across multiple backbones and visual tool-use benchmarks.
2 Related Work
Prior multimodal compression methods mainly reduce visual tokens or shorten generated text, leaving accumulated reasoning redundancy undiagnosed. SPARE instead prunes stale self-generated text to improve visual grounding without auxiliary models.
- Visual-token pruning reduces visual-stream redundancy but can weaken image evidence that is already under-attended in deep layers.
- Concise-reasoning and generic summarization shorten text or compress history but do not diagnose which historical reasoning segments remain functionally necessary.
- Training, decoding, and attention-reweighting methods address language-prior bias while leaving competing textual redundancy intact and often requiring extra inference or auxiliary data.
- SPARE operates at test time, requires no auxiliary model, and improves grounding by pruning stale self-generated text that competes with visual evidence.
3 Preliminaries
The paper models a VLM agent as an iterative process that generates reasoning, invokes tools, incorporates results, and eventually answers. OPSD supplies a diagnostic principle for comparing the same model under related contexts.
- The VLM receives a question and image tokens, then performs multi-step reasoning with external tools before producing a final response.
- Each reasoning–action segment contains a free-form reasoning span and either a tool invocation or a final answer.
- Tool results are appended to the message history, allowing interaction to continue until the model emits a final response.
- Conventional off-policy distillation can create exposure bias because its training prefixes differ from student-generated inference prefixes.
- In OPSD, the same model supplies teacher and student distributions, eliminating the need for an external teacher.
- SPARE uses OPSD only diagnostically, measuring distributional shifts between related contexts to assess whether summaries cover historical reasoning.
4 Method
SPARE probes whether a compact task-state summary covers historical reasoning by replaying segments with and without summary conditioning. Reverse-KL sensitivity then drives conservative segment pruning and evidence-preserving reconstruction.
- 4 Method: SPARE is a post-hoc method that compresses linguistic scaffolding while preserving the task state underlying multi-step visual reasoning.
- Adaptive summary trigger: The adaptive trigger invokes SPARE when the agent generates a compact summary containing the question, tool calls, visual evidence, and remaining uncertainty.
- Summary-conditioned replay: Summary-conditioned replay compares the same model under contexts with and without the compact summary to test whether the summary covers candidate segments.
- Summary-conditioned replay: A small continuation-distribution shift indicates summary coverage, whereas a larger shift indicates information in the original reasoning that should be preserved.
- Top-K reverse-KL coverage score: High reverse-KL values mark residual information not covered by the summary, while low values indicate that the summary sufficiently explains the token context.
- Segment-level pruning: Segment token positions map token-level sensitivity back to historical segments for pruning decisions.
- Segment-level pruning: A segment is retained when it contains at least κ high-sensitivity tokens; otherwise, low-residue content is removed.
- Segment-level pruning: The count rule preserves sparse modality-critical evidence, such as OCR strings, coordinates, bounding boxes, or candidate labels, that averaging could dilute.
5 Experiments
Experiments evaluate SPARE across visual tool-use benchmarks, backbones, pruning baselines, summarizer fine-tuning, and triggering policies. SPARE generally preserves or improves accuracy while substantially pruning reasoning history.
- Experimental Settings: SPARE is evaluated on several multi-step visual tool-use benchmarks and three vision–language backbones under a fixed tool-use interface.The benchmarks require reasoning, tool invocation, and intermediate visual observations.
- Experimental Settings: The comparison includes full-trace, direct-answer, complete reasoning deletion, and other inference strategies under an identical tool-use harness.Pruning is measured as the percentage of reasoning-side history tokens removed relative to Tool Baseline.
- Main Results: 63.70% reasoning-token pruning accompanies an average-accuracy increase from 44.30% to 49.19% for SPARE on Qwen3-VL-30B.On Qwen3-VL-8B, SPARE preserves comparable performance, 53.47% versus 54.27%, with 37.89% pruning.
- Main Results: 64.58% pruning lets SPARE nearly match Nemotron’s overall Full Trace accuracy while improving Full Trace on GTA, B-Jigsaw, and VTB.The benchmark results also show Direct Answer outperforming Full Trace for Nemotron on all applicable benchmarks.
- SFT for Summarization Capability: SFT-based summarizer fine-tuning enables more aggressive pruning and improves task accuracy on both VisualToolBench and MNMS.The result supports stronger summaries expanding the safe-to-prune region without harming downstream decisions.
- Ablation Study: Model-selected summarization achieves 42.37% accuracy and 62.42% pruning, while increasing κ to 3 slightly raises pruning but lowers accuracy.The study adopts (τ, κ) = (0.2, 2) because κ = 1 and κ = 2 produce identical results and the setting is locally stable.
6 Conclusion
The paper identifies textual debt as a failure mode in multi-step MLLM agents and proposes selective reasoning pruning to preserve visual evidence. SPARE improves performance and reduces reasoning-token usage while redirecting attention toward images.
- Conclusion: Textual debt arises when accumulated self-generated reasoning dominates context and weakens reliance on visual evidence.The paper links this failure mode to changing reasoning utility and stale linguistic assumptions.
- Conclusion: SPARE selectively removes redundant reasoning while preserving visual evidence through an OPSD KL-guided framework.The method is designed to reduce textual dominance and redirect attention to images.
- Conclusion: Experiments show that SPARE improves task performance, reduces reasoning-token usage, and restores attention to visual evidence.The paper presents inference-time selective forgetting as an effective mechanism for long-context multimodal reasoning.
A Additional Preliminaries
The preliminaries formalize multi-step tool use, on-policy self-distillation, and cross-modal attention measurement. These definitions establish the trajectory structure and diagnostic quantities used by the method.
- A.1 Multi-Step Tool Use in MLLMs: A vision-language model answers questions using image tokens and may query external tools over multiple interaction steps.Each step conditions on the current message history and either invokes a tool or emits a final response.
- A.1 Multi-Step Tool Use in MLLMs: Each assistant message contains a free-form reasoning span and an action block, with past reasoning–action segments forming the trajectory history.Tool results are appended to the history before the next interaction step.
- A.2 On-policy self-distillation: On-policy distillation applies the loss to trajectories sampled from the student, reducing mismatch with student-generated inference prefixes.This contrasts with off-policy distillation on fixed teacher or ground-truth sequences.
- A.2 On-policy self-distillation: In on-policy self-distillation, the teacher and student share the same model, eliminating the need for an external teacher.The teacher distribution is treated as a fixed target while gradients update the student parameters.
- A.3 Self-attention over modalities: The visual attention ratio summarizes how strongly response-generation tokens attend to image tokens across decoder layers.It is computed from attention directed toward visual keys and averaged over heads and textual query tokens.
- A.3 Self-attention over modalities: Visual attention is empirically high in early layers but decays sharply with depth, leaving deep layers focused mainly on textual tokens.This observation motivates monitoring visual reliance in multimodal reasoning.
B Additional Experimental Details
SPARE evaluates pruning over persistent reasoning-side history while leaving tool interactions, observations, images, and final answers unchanged. Its accounting measures original versus retained reasoning tokens, with transient summary probes excluded from persistent history.
- Experimental setup: SPARE leaves tool-call JSON, tool observations, images, final answers, and transient summary probes unchanged or excluded from persistent reasoning accounting.Only accumulated assistant reasoning is retained or reconstructed.
- Token accounting: Each reasoning segment is counted exactly once using its original text and final persistent representation, regardless of repeated summary inspections.The final representation may preserve, delete, or reconstruct the segment as structured visual evidence.
- Runtime: At each pruning event, SPARE generates a task-state summary and performs two forward replays to compute the summary-conditioned KL signal.This creates temporary inference overhead, while adaptive triggering avoids the cost on short trajectories.
C.1 Controlled Diagnostic of Textual Interference
The controlled diagnostic isolates cases where stale early hypotheses conflict with later corrective tool evidence. SPARE outperforms random deletion in reducing stale-answer copying while preserving the visual and tool evidence.
- Controlled construction: The diagnostic constructs audited GTA cases with plausible but incorrect early answers followed by corrective tool observations.Questions, images, tool calls, action blocks, and observations remain fixed across conditions.
- Conditions: The comparison includes Full Trace, random deletion of one eligible segment, and automatic SPARE pruning.SPARE preserves the tool interaction and visual evidence, and cases without eligible pruning remain included.
- Metrics: Forced-choice accuracy measures selection of the later evidence-supported answer, while stale-copy rate measures repetition of the obsolete textual hypothesis.A lower stale-copy rate indicates less reliance on obsolete reasoning when it conflicts with later evidence.
- Results: For Qwen3-VL-8B, SPARE raises forced-choice accuracy from 63.64% to 90.91% and lowers stale-answer copying from 36.36% to 9.09%.For Qwen3-VL-30B-A3B, accuracy rises from 54.55% to 81.82%, while SCR falls from 45.45% to 18.18%.
- Interpretation: Random-1 yields smaller improvements than SPARE, indicating that targeted selection matters beyond shortening the reasoning history.Free-form answer accuracy remains unchanged between Full Trace and SPARE.
- Scope: This diagnostic supports the textual-debt mechanism but is not a general comparison with every reasoning-compression strategy.It isolates a specific conflict between stale text and later evidence.
C.2 Attention Visualization After Pruning
Attention visualization examines whether pruning changes the balance between textual context and image evidence. Across nearly all decoder layers, SPARE increases attention to image tokens after pruning.
- Visualization: The visualization averages text-to-image attention across 10 tool-use trajectories and compares attention before versus after pruning.Blue and orange encode the two conditions, while shading shows the gain.
- Finding: Pruning summary-covered reasoning consistently increases attention to image tokens across decoder layers.The shift provides evidence that suppressing textual dominance restores visual evidence’s role in downstream reasoning.
D Limitations
SPARE is designed for multi-step multimodal agents with explicit reasoning and tool-use histories, and its broader applicability remains unresolved. The method also adds summary-generation and context-replay computation, although adaptive triggering limits this cost on short trajectories.
- Scope: SPARE’s application to single-turn or unstructured agents is less direct because it targets multi-step multimodal tool-use histories.The stated design scope is explicit reasoning and tool-use agents.
- Overhead: Summary generation and context replay introduce additional computation at test time.Adaptive triggering avoids this cost on short trajectories, and the pruned context can be reused in subsequent steps.
- Future scope: Extending SPARE to additional modalities and agent architectures, with more comprehensive end-to-end latency evaluation, remains future work.The paper reports consistent results across three backbones and five benchmarks under one pruning configuration.