The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gating

Landi He; Xiaoyu Yang; Lijian Xu

Source-linked AI summary

The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gating

Landi He, Xiaoyu Yang, Lijian Xu

arXiv:2603.07135v1Mar 7, 2026cs.CV

TL;DR

Visual tokens create substantial VLM inference cost despite redundancy, motivating pruning under a fixed capacity budget. AutoSelect learns this allocation with lightweight scoring and denoising modules attached to a frozen VLM, retaining 96.5% of full-model accuracy at 88.9% pruning while transferring across backbones.

Problem
Visual tokens can dominate VLM computation, while existing pruning typically selects tokens using local attention or similarity signals rather than globally allocating capacity under a fixed budget.
Method
AutoSelect attaches a Scorer and Denoiser to a frozen VLM and trains capacity allocation with next-token prediction, variance-preserving noise gating, and differentiable Top-K selection.
Results
96.5% of full-model performance is retained at 88.9% pruning on LLaVA-1.5-7B with 0.69 ms selection overhead, while the method generalizes to LLaVA-NEXT and Qwen2.5-VL.
Takeaways & Limitations
Capacity-constrained allocation preserves high VLM performance under aggressive token reduction and transfers across higher-resolution and architecturally distinct backbones.

Abstract

from arXiv · show

Visual tokens dominate inference cost in vision-language models (VLMs), yet many carry redundant information. Existing pruning methods alleviate this but typically rely on attention magnitude or similarity scores. We reformulate visual token pruning as capacity constrained communication: given a fixed budget K, the model must allocate limited bandwidth to maximally preserve visual information. We propose AutoSelect, which attaches a lightweight Scorer and Denoiser to a frozen VLM and trains with only the standard next token prediction loss, without auxiliary objectives or extra annotations. During training, a variance preserving noise gate modulates each token's information flow according to its predicted importance so that gradients propagate through all tokens; a diagonal attention Denoiser then recovers the perturbed representations. At inference, only the Scorer and a hard top-K selection remain, adding negligible latency. On ten VLM benchmarks, AutoSelect retains 96.5% of full model accuracy while accelerating LLM prefill by 2.85x with only 0.69 ms overhead, and transfers to different VLM backbones without architecture-specific tuning. Code is available at https://github.com/MedHK23/AutoSelect.

1 Introduction

AutoSelect reframes visual token pruning as capacity-constrained representation learning, allocating a fixed information budget across tokens rather than simply discarding them. Its noise-gated, differentiable training framework uses lightweight modules with a frozen VLM and standard next-token prediction loss.

Motivation: Visual-token redundancy creates a major VLM bottleneck because self-attention still processes low-contribution tokens.Attention is often concentrated on a small subset, while subsequent layers compute over all visual tokens.
Motivation: Existing pruning methods commonly use local signals such as attention magnitude, similarity scores, or predefined schedules.These approaches frame pruning primarily as identifying and discarding less important tokens.
Capacity-Constrained Formulation: AutoSelect models the encoder–LLM interface as a bandwidth-limited channel whose fixed budget K constrains information capacity rather than token count.During training, all tokens remain present while their effective information throughput is continuously modulated by importance.
Method: The method combines a Scorer and Denoiser with a frozen VLM, attenuating low-scoring tokens through a continuous bottleneck during training.The Denoiser remaps perturbed tokens toward the distribution expected by the frozen LLM while operating independently on each token.
Method: Variance-preserving noise gating and Soft Top-K selection provide gradient flow during training while converging to hard Top-K selection at inference.The framework is trained with the standard next-token prediction loss without auxiliary losses, external annotations, or base-VLM modifications.
Results: 96.5% of full-model performance is retained at 88.9% pruning on LLaVA-1.5-7B with 0.69 ms pruning-module overhead.The method also generalizes to higher-resolution and architecturally distinct VLMs.

2 Related Work

Prior VLM efficiency work reduces visual-token cost through pruning, merging, calibration, staged schedules, or deeper in-LLM selection. These approaches expose trade-offs among speed, semantic information, annotation requirements, and backbone assumptions.

VLM Efficiency Setting: VLMs increasingly face rising visual-token counts from high-resolution, multi-image, arbitrary-aspect-ratio, and video inputs.This growth amplifies LLM self-attention cost and KV-cache overhead.
Token Reduction: Token reduction methods commonly prune or merge tokens, with VLM work targeting the LLM prefill stage where long visual prefixes increase computation and memory.Plug-and-play methods can prune immediately after the vision encoder.
Token Reduction: Calibration-based and staged methods balance local feature preservation against downstream effects, while in-LLM pruning trades earlier computation for potentially better task retention.The approaches differ in where semantic mixing occurs and how much early computation is preserved.
Trade-offs: Pruning location creates an accuracy–speed trade-off: late pruning uses higher-level semantics, whereas early pruning maximizes savings but may discard fine-grained evidence.This boundary is especially relevant for text-rich or localization-sensitive queries.
Learned Selection: Learned selection methods can depend on bounding-box annotations or adaptive modules inserted between LLM layers.These design choices may constrain open-vocabulary generalization or alter the location of computation.

3 Methodology

AutoSelect inserts lightweight Scorer and Denoiser modules between a frozen vision encoder and projector, learning token importance under a fixed budget while preserving differentiable training. Soft Top-K and variance-preserving noise constrain token capacity during training, whereas inference retains only hard top-K tokens with original positions.

Framework overview: AutoSelect places a learnable Scorer and Denoiser between the frozen vision encoder and projector, optimizing only these modules with the standard language-modeling objective.The original VLM parameters remain frozen, and no auxiliary losses or hand-crafted features are required.
Inference: At inference, AutoSelect discards noise injection and the Denoiser, then applies hard top-K selection while preserving original token position indices.Preserved indices allow the LLM’s rotary position embeddings to retain spatial token locations, and visual-only scores can be reused across dialogue turns.
Learnable token scoring: The Scorer assigns per-token importance scores and uses differentiable Soft Top-K to impose a fixed bandwidth budget while retaining all tokens during training.Soft Top-K produces a budget-constrained soft assignment that becomes increasingly binary as its temperature is annealed.
Capacity-constrained gating: During training, variance-preserving noise modulates each token according to its importance, allowing gradients to flow through the full sequence without hard deletion.High-score tokens preserve the original signal, while low-score tokens approach isotropic Gaussian noise; variance-preserving coefficients maintain feature scale.
Capacity-constrained gating: The Denoiser maps noise-perturbed representations toward the frozen LLM’s expected input distribution using diagonal attention that prevents cross-token information leakage.Each token attends only to itself, yielding independent per-token transformations.

4 Experiments

AutoSelect is evaluated across three VLM backbones, ten benchmarks, varying token budgets, and LLM-free classification. It achieves strong performance retention while reducing prefill latency, and ablations examine its selection quality and capacity-constrained training mechanisms.

Experimental Setup: AutoSelect is evaluated on LLaVA-v1.5-7B, LLaVA-NEXT-7B, and Qwen2.5-VL-7B across ten standard VLM benchmarks.The baselines are grouped by whether pruning occurs before or inside the LLM.
Main Benchmark Results: At 64 retained tokens, AutoSelect reaches 96.5% average retention on LLaVA-1.5-7B under 88.9% pruning.At 128 tokens, it surpasses all baselines in average retention; at 192 tokens, it is slightly below PRUNESID in average accuracy.
Main Benchmark Results: AutoSelect achieves 96.1% average retention on LLaVA-NEXT-7B while retaining 320 tokens, outperforming HoloV by 0.4%.The setting uses 2,880 original visual tokens and an 88.9% reduction.
Main Benchmark Results: AutoSelect outperforms all baselines across three pruning rates on Qwen2.5-VL-7B, whose visual token count varies with native image resolution.Its per-token Scorer requires neither a fixed sequence length nor assumptions about grid layout.
LLM-Free Evaluation: In LLM-free ImageNet-1K classification, AutoSelect surpasses FastV and HoloV at every budget in both Top-1 and Top-5 accuracy.At K=6, representing 3% of tokens, it leads HoloV by roughly 10 percentage points in Top-1 accuracy.

5 Conclusion

AutoSelect reframes visual token pruning as capacity-constrained representation learning, using differentiable information allocation instead of outright token deletion. It retains strong accuracy, adds minimal selection overhead, and transfers across VLM backbones without architecture-specific modification.

AutoSelect modulates per-token information throughput with variance-preserving noise, turning discrete pruning into continuous optimization trained only with next-token prediction loss.A diagonal-attention Denoiser prevents information leakage during training and is removed at inference.
96.5% of full-model accuracy is retained at 88.9% token pruning on LLaVA-1.5-7B.
0.69 ms of selection overhead is incurred while the method transfers to LLaVA-NEXT and Qwen2.5-VL without architecture-specific modification.
AutoSelect consistently outperforms existing methods across all evaluated settings, supporting learned capacity allocation over heuristic pruning criteria.