PaperlayerView exact revision on arXiv

Source-linked AI summary

An AI4AI Framework for Visual Token Pruning

Zhen Liu, Wenli Huang, Wei Song, Yuhan Liu, Zhiqin Yang, Jingwen Fu

arXiv:2608.07193v1cs.LGcs.CV

TL;DR

Handcrafted visual-token pruning policies require costly expertise and trial and error as objectives, budgets, and architectures diversify. AutoPrune uses LLM-driven residual policy search through TPDSL, improving the accuracy–efficiency trade-off across benchmarks and backbones while retaining over 99% performance after removing 94.4% of visual tokens.

  • Problem

    Visual-token pruning relies heavily on handcrafted policies, making manual design increasingly difficult as objectives, budgets, and MLLM architectures diversify.

  • Method

    AutoPrune uses TPDSL to represent LLM-designed pruning candidates as constraint-aware residual modifications to a strong base policy.

  • Results

    94.4% visual-token reduction preserves more than 99% of full-token performance, while reducing FLOPs by 9.9× and prefill latency by 6.4×.

  • Takeaways & Limitations

    Across benchmarks and MLLM backbones, AutoPrune improves the accuracy–efficiency trade-off over representative handcrafted pruning methods, especially under tighter token budgets.

  • Takeaways & Limitations

    AutoPrune depends on the expressiveness of TPDSL and the reliability of its task evaluator.

Abstract

from arXiv · show

Visual-token pruning can substantially reduce the inference cost of multimodal large language models (MLLMs), yet existing methods largely rely on fixed, handcrafted heuristics and costly expert trial and error. As pruning objectives, budgets, and model architectures diversify, manually navigating the expanding design space becomes increasingly difficult. This paper aims to build an AI4AI framework for visual-token pruning by addressing a natural question: Can large language models automatically design effective visual-token reduction algorithms? Although LLMs possess broad algorithmic knowledge and strong reasoning capabilities, translating such general knowledge into effective solutions for a specialized task remains nontrivial. We argue that the key lies in designing an appropriate search-state representation that connects the internal knowledge of LLMs with the structural requirements and constraints of visual-token pruning. Based on this insight, we propose AutoPrune, a training-free framework for LLM-driven visual-token pruning policy design. At its core, AutoPrune introduces a Token Pruning Domain-Specific Language (TPDSL) comprising 131 reusable atoms for budget control, token scoring, selection constraints, and token reassembly. A key property of TPDSL is that it represents each search state as a residual modification of a strong base policy. This residual formulation narrows the search space and directs the LLM's attention toward the policy components that are most consequential for performance. Experiments on 14 multimodal benchmarks and three MLLM backbones demonstrate the effectiveness, efficiency, and transferability of AutoPrune. Even when removing 94.4% of visual tokens, AutoPrune preserves more than 99% of full-token performance while reducing FLOPs by 9.9x and prefill latency by 6.4x.

Introduction

AutoPrune is a training-free framework that uses TPDSL and residual search around a strong base policy to let LLMs design visual-token pruning policies. Across 14 multimodal benchmarks and three MLLM backbones, it achieves substantial token reduction with limited performance loss and lower inference cost.

  • Motivation: Existing pruning methods rely on handcrafted criteria, making policy design increasingly difficult as objectives, token budgets, and MLLM architectures diversify.Designing these policies requires substantial domain expertise and costly trial and error.
  • Motivation: Naively representing candidates as complete standalone Python programs forces LLMs to navigate a vast design space while rediscovering existing mechanisms and satisfying strict pruning constraints.The paper identifies search-state representation as the central challenge in making general LLM algorithmic knowledge actionable for visual-token pruning.
  • Method: AutoPrune introduces TPDSL, a training-free domain-specific language with 131 reusable atoms for budget control, token scoring, selection constraints, and token reassembly.The framework requires no fine-tuning of the target MLLM.
  • Method: AutoPrune formulates policy design as residual search around a strong base policy, using structured search states, evaluator-guided search, and quota-constrained residual execution.This residual formulation avoids designing each candidate policy from scratch and narrows the search toward consequential policy components.
  • Results: 94.4% visual-token reduction preserves more than 99% of full-token performance on both LLaVA-1.5-7B and LLaVA-NeXT-7B.In the 320-token efficiency setting on LLaVA-NeXT-7B, AutoPrune reduces FLOPs by 9.9× and prefill latency by 6.4×.

Related Work

Prior visual-token pruning methods reduce MLLM inference cost through learned selection, latency-aware pruning, token squeezing, or other largely handcrafted policies requiring task-specific tuning. AutoPrune addresses this design challenge by using structured residual search with TPDSL and evaluator-in-the-loop validation to produce executable, budget-compliant, transferable policies.

  • Visual-token pruning: Visual-token pruning reduces MLLM inference cost by removing or compressing redundant visual tokens, following earlier efficient Vision Transformer work on selection, latency-aware pruning, and token squeezing.The passage cites Kim et al. 2022, Kong et al. 2022, Wei et al. 2023, and Liu et al. 2024c, alongside MLLM methods including FastV, PyramidDrop, and SparseVLM.
  • Visual-token pruning: Existing pruning policies are largely handcrafted and require task-specific tuning, whereas AutoPrune automates policy design through structured residual search around a strong base policy.The residual search formulation targets policy-design automation rather than manual tuning.
  • LLM-driven algorithm design: LLMs have been used as architecture generators, mutation operators, and evaluator-guided program-search systems for algorithm and mathematical discovery.Examples include EvoPrompting, LLMatic, FunSearch, and AlphaEvolve.
  • LLM-driven algorithm design: Directly applying LLMs to visual-token pruning is difficult because candidate policies must satisfy strict budget, tensor, index, and numerical constraints.AutoPrune uses TPDSL to represent each search state as a constrained residual modification to a base policy.
  • LLM-driven algorithm design: Evaluator-in-the-loop validation and selection enable AutoPrune to search for executable, budget-compliant, and transferable pruning policies.These mechanisms are combined with TPDSL’s constrained residual representation.

Method

AutoPrune frames visual-token pruning as constrained LLM-driven policy design over TPDSL residual modifications to a strong base policy. TPDSL structures, validates, searches, and transfers pruning policies while bounded residual refinement preserves most base-policy selections and revises uncertain tokens.

  • Residual policy representation: AutoPrune represents each candidate as a TPDSL residual modification to a strong base policy, narrowing the search space instead of generating complete pruning programs.The residual search state explicitly encodes pruning operations and execution constraints.
  • LLM-driven search: A pre-trained LLM iteratively proposes, validates, and evaluates TPDSL residual states using accumulated search-history feedback, then selects the highest-scoring valid candidate.Invalid candidates are excluded from task evaluation, while observations promote effective compositions and discourage invalid or low-performing designs.
  • TPDSL design: TPDSL defines α = (B, S, C, R) for budget control, token scoring, selection constraints, and token reassembly.The four components cover budget instantiation, quality-signal fusion, admissible selection behavior, and reassembly.
  • TPDSL design: 131 instantiated atoms organize TPDSL into nine functional groups with standardized interfaces that constrain budget-, shape-, and index-compatible compositions.The atoms cover token sources, budgets, scoring, similarity kernels, score fusion, candidate pools, constraints, reassembly, token processing, and safety checks.
  • Residual refinement: The selected policy performs bounded residual exchange, preserving at least rmin base-policy tokens while replacing only uncertain selections, then restores token ordering and discarded-token information when specified.The refined subset maintains the dominant base-policy structure while correcting a small number of uncertain token selections.
  • Policy transfer: AutoPrune re-materializes the selected TPDSL state for a target budget, task, or MLLM backbone by updating only budget- and model-dependent fields, enabling transfer without additional LLM-driven search.Each transferred policy is validated before deployment.

Experiments and Results

AutoPrune delivers strong accuracy retention and efficiency gains across multimodal benchmarks, token budgets, and MLLM backbones. Ablations show that residual reference anchoring and limited token exchange are central to its performance, while improvements remain robust across proposers and base policies.

  • Benchmarks: AutoPrune evaluates 14 image-based multimodal benchmarks spanning visual question answering, scientific reasoning, hallucination, text-rich, and diagram-oriented understanding.Experiments follow each benchmark’s official data splits, evaluation protocols, and metrics.
  • Comparison with State-of-the-Art Methods: 94.4% token removal yields aggregate scores of 63.2 and 65.2 on LLaVA-1.5-7B and LLaVA-NeXT-7B, retaining 99.7% and 99.9% of full-token performance.AutoPrune outperforms CDPruner by 3.2 points and 2.4 points on the respective backbones.
  • Efficiency on LLaVA-NeXT-7B: 9.9× lower FLOPs and 6.4× lower prefill latency accompany a KV-cache reduction from 1440.0 MB to 160.0 MB with 320 retained visual tokens.These measurements use LLaVA-NeXT-7B on a single NVIDIA RTX 3090 GPU.
  • Transfer to Advanced MLLM Architectures: 6.1 percentage points increase relative performance from 74.9% to 81.0% at 128 tokens on Qwen2.5-VL-7B, with gains across all evaluated token budgets.The searched TPDSL state transfers to a distinct visual-token architecture and performs especially well on text-rich visual reasoning tasks.
  • Effect of TPDSL Components: Removing reference anchoring reduces MME from 1413.46 to 1223.29, while removing multi-score fusion lowers it to 1391.70.The ablation supports residual modification of a base policy and fusion of complementary relevance, saliency, spatial, redundancy, and contrast signals.
  • Effect of the Residual Exchange Quota: 40.46 MME points is the improvement from setting qe = 2, whereas replacing all 32 base-policy tokens reduces the score to 1217.88.A small exchange quota preserves most base-policy selections while correcting a limited number of uncertain tokens.
  • Generalization across Base Token-Pruning Strategies: 145.40–22.46 MME points are the gains AutoPrune achieves over all five strategy-only baselines, including 145.40 for PruMerge+ and 127.80 for VisionZip.The consistent improvements indicate that residual search is not tied to a particular base policy.

Conclusion

AutoPrune makes LLM-driven visual-token pruning practical through structured residual search, improving the accuracy–efficiency trade-off across benchmarks and MLLM backbones. Its effectiveness depends on TPDSL expressiveness and evaluator reliability, motivating richer representations and broader validation.

  • Conclusion: AutoPrune uses TPDSL to express executable, budget-compliant, safety-validated modifications to a strong base policy instead of unconstrained pruning code.This residual design preserves reliable prior selections while enabling limited, task-adaptive token replacement.
  • Conclusion: AutoPrune improves the accuracy–efficiency trade-off over representative handcrafted pruning methods across multiple benchmarks and MLLM backbones, with larger gains under tighter token budgets.The results support constraint-aware residual modifications as a more effective search formulation than free-form code.
  • Conclusion: AutoPrune depends on TPDSL search-space expressiveness and task-evaluator reliability, motivating richer search-state representations, adaptive evaluation, and broader validation.Future validation will span additional MLLM architectures and multimodal tasks.

Supplementary Material … Selected TPDSL State

The supplementary material details AutoPrune’s 131-atom TPDSL, evaluator-in-the-loop search, materialization and safety validation, and selected residual policy. The selected state preserves at least 30 of 32 base-policy tokens while allowing at most two replacements per input.

  • A Additional Method Details: The supplementary material covers TPDSL organization, LLM-driven policy design, and the materialized state selected under the source setting.It describes the 131 atoms, complete evaluator-in-the-loop procedure, and selected TPDSL state.
  • TPDSL Atom Library: Each TPDSL search state is a residual modification of a base policy, α = (B, S, C, R), spanning budget control, token scoring, selection constraints, and token reassembly.The library contains 131 instantiated atoms organized into nine functional groups.
  • TPDSL Atom Library: 81 variants belong to reassembly and token processing, covering anchor assignment, residual aggregation, token ordering, and shape-preserving output.Similarity kernels, score-fusion operators, and safety checks require fewer variants because they implement more standardized operations.
  • TPDSL Atom Library: Materialization binds selected atoms to runtime implementations, resolves parameters, checks compatibility, and constructs scoring, selection, exchange, and reassembly operators.Standardized TPDSL interfaces support re-instantiation under different token budgets and MLLM backbones without modifying the searched policy structure.
  • TPDSL Atom Library: Candidates are safety-validated for budget consistency, token-index validity and uniqueness, numerical stability, tensor compatibility, and deterministic execution before evaluation.Invalid candidates are excluded from task evaluation, with optional fallback to base-policy selection.
  • Procedure of LLM-Driven Policy Design: Each search round proposes residual states, materializes and validates candidates, evaluates valid policies, and records successes and failures as feedback.The history retains high-performing combinations, score trends, and recurrent validation failures; LLM calls occur only during offline design.
  • Selected TPDSL State: Under LLaVA-1.5-7B on MME with K0 = 32 visual tokens, the selected Full TPDSL state modifies the CDPruner base policy using multi-signal scoring, diversity-aware selection, and bounded residual exchange.The configuration preserves most base-policy tokens while revising only a small number of uncertain selections.
  • Selected TPDSL State: At least 30 of 32 base-policy tokens are preserved, while at most two replacements per input are permitted.This bounded-correction configuration leaves the base policy as the dominant token-selection structure.

B Experimental Settings … Evaluation Metrics and Efficiency Protocol

AutoPrune is evaluated across three MLLM backbones with a fixed, training-free search configuration and validated executable policies. Performance uses benchmark-specific official metrics plus aggregate and relative scores, while efficiency is measured under identical single-GPU settings.

  • MLLM Backbones: AutoPrune is evaluated on LLaVA-1.5-7B, LLaVA-NeXT-7B, and Qwen2.5-VL-7B, spanning differences in resolution handling, visual-token organization, and language-model architecture.
  • MLLM Backbones: Evaluation follows Zhang et al. (2025) protocols for LLaVA models, while Qwen2.5-VL-7B retains its native visual-token architecture and reproduces CDPruner.
  • Search Configuration and Implementation Details: AutoPrune is training-free with frozen MLLM parameters; policy design defaults to LLaVA-1.5-7B, MME, K0 = 32, Qwen-Plus, and CDPruner as base policy πP.
  • Search Configuration and Implementation Details: 10 rounds with five candidate search states each produce 50 candidates, which are validated for budget consistency, index validity, numerical stability, output-shape compatibility, and deterministic execution.
  • Search Configuration and Implementation Details: rmin = 30 and qe = 2 preserve at least 30 base-policy tokens under the 32-token source budget, allowing at most two searched replacements.
  • Search Configuration and Implementation Details: The selected TPDSL state is evaluated at the source setting, then re-instantiated for other token budgets and MLLM backbones without additional LLM-driven search.
  • Evaluation Metrics and Efficiency Protocol: Evaluation reports each benchmark’s official metric, aggregate score Acc., relative performance Rel., and efficiency metrics including retained visual tokens, FLOPs, latency, KV-cache size, and peak GPU memory.MME is normalized before aggregation, with cMME = 20 for the LLaVA benchmark suite and cMME = 28 for the Qwen2.5-VL-7B benchmark suite; relative performance compares against the full-token model.
  • Evaluation Metrics and Efficiency Protocol: All methods use identical checkpoints, token budgets, and inference configurations on a single NVIDIA RTX 3090 GPU, with best and second-best pruning results marked per budget.

C Additional Experimental Results

This section presents complete benchmark results, cross-budget and cross-backbone transfer, comparisons with alternative LLM-driven search strategies, and search-stability analyses. AutoPrune reuses a TPDSL state searched at K0 = 32 on LLaVA-1.5-7B, re-instantiating only budget- and architecture-dependent fields elsewhere.

  • C Additional Experimental Results: The section reports complete per-benchmark results, cross-budget transfer, alternative LLM-driven search comparisons, and random-seed search-stability analyses.
  • C Additional Experimental Results: AutoPrune's default TPDSL search state is selected on LLaVA-1.5-7B at source budget K0 = 32.
  • C Additional Experimental Results: For other token budgets and MLLM backbones, AutoPrune re-instantiates only budget- and architecture-dependent fields without additional LLM-driven search.

Results on LLaVA-1.5-7B

On LLaVA-1.5-7B, AutoPrune outperforms representative training-free pruning methods across transferred visual-token budgets, with larger gains under tighter budgets. At 32 tokens, it reaches an aggregate score of 63.2 while preserving 99.7% of full-token performance.

  • Evaluation setup: The comparison uses the same model checkpoint, benchmark splits, prompting, decoding configurations, and visual-token budgets for all methods.Compared methods include FastV, PyramidDrop, SparseVLM, PruMerge+, TRIM, VisionZip, DART, DivPrune, and CDPruner.
  • Cross-budget comparison: AutoPrune improves the aggregate score over CDPruner by 2.3, 2.5, and 3.2 points at 128, 64, and 32 retained tokens, respectively.The corresponding relative-performance gains are 3.3, 3.9, and 5.4 percentage points.
  • Cross-budget transfer: AutoPrune searches only at the 32-token source budget, then re-instantiates the selected TPDSL state at 64 and 128 tokens without additional search.Consistent gains at transferred budgets show that the selected search state generalizes beyond its source budget.
  • Tight-budget performance: At 32 retained tokens, AutoPrune achieves an aggregate score of 63.2 and preserves 99.7% of full-token performance.The advantage becomes larger under tighter token budgets.

Results on LLaVA-NeXT-7B … Search Stability across Random Seeds

AutoPrune transfers its TPDSL search state across LLaVA backbones and token budgets without additional search, outperforming the CDPruner baseline and remaining stable across search strategies and random seeds. Its constrained residual-search formulation consistently produces effective, executable pruning policies.

  • Results on LLaVA-NeXT-7B: The LLaVA-1.5-7B search state transfers to LLaVA-NeXT-7B by re-instantiating budget-dependent fields without additional LLM-driven search.The transferred state is evaluated at 640, 320, and 160 retained visual tokens.
  • Results on LLaVA-NeXT-7B: At 160 retained tokens, AutoPrune preserves 99.9% of full-token performance on LLaVA-NeXT-7B while improving aggregate score over CDPruner by 2.4 points.The aggregate-score improvements are 1.7 and 1.9 points at 640 and 320 tokens, with relative-performance gains of 2.4, 3.0, and 3.9 percentage points.
  • Cross-Budget Policy Transfer: The selected 32-token search state generalizes to 16-, 64-, and 128-token budgets through re-instantiation without additional search.The 32-token result uses the selected source policy directly, while other budgets modify its budget-dependent fields.
  • Cross-Budget Policy Transfer: At 16 retained tokens, AutoPrune preserves 96.6% of full-token performance, while aggregate score rises from 61.2 to 64.8 as the budget increases to 128 tokens.Relative performance increases from 96.6% to 102.3% across the same budget range, without repeated policy design.
  • Comparison with Different Search Strategies: Under identical starting policies, evaluators, budgets, and search configurations, all compared strategies improve over CDPruner, with AutoPrune achieving the best result.The comparison includes OpenEvolve and an EvoPrompting-style implementation at a 32-token budget using 10 rounds with five candidates per round.
  • Search Stability across Random Seeds: All four random-seed runs produce 50 valid candidates under a 10×5 search budget, demonstrating consistently executable and budget-compliant TPDSL policies.Each run evaluates 50 candidate policies under the same experimental setting.
  • Search Stability across Random Seeds: All four seeds recover the same best MME score of 1413.46, with candidate-level means from 1408.23 to 1408.94 and minimum scores above 1406.82.These results indicate that Full TPDSL does not depend on a lucky high-performing proposal and that gains are supported by its structured, constrained residual-search design.
Loading 2608.07193v1…