PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Models

• At 71.3% visual token reduction, PACT incurs a 1.4% performance drop versus 4.4% for previous state-of-the-art methods at equal reduction; 50% reduction brings negligible performance loss.Source: §1 · p. 1
• PACT's token-importance metric avoids attention scores, making it compatible with FlashAttention; its pruning and clustering modules require no additional training.Source: §1 · pp. 1–2
• Distance Bounded Density Peak Clustering groups tokens within a predefined distance threshold, while PACT recovers pruned tokens close to clusters before merging.Source: §1 · p. 1
• Combining pruning with clustering-based merging surpasses either technique alone, and PACT outperforms previous and concurrent works [3] [6] [9] [30] [44].Source: §1 · p. 2

2.1. Visual language models

• High-resolution models still use over 10,000 tokens (Qwen-VL2); adjacent-token merging preserves structure but does not address token irrelevance or redundancy.Source: §2.1 · p. 2

2.2. Visual token reduction

• EViT [29] merges [CLS]-attention-irrelevant tokens, ToME [6] iteratively merges similar tokens, and LaVIT [21] trains a mask to select retained tokens.Source: §2.2 · p. 2
• PruMerge [44] and HiRED [3] rely on ViT/[CLS] assumptions and face pre-merging attribution issues; TRIM [46] uses text, FastV [9] attention scores, and VTW [30] prunes late.Source: §2.2 · p. 2
• Prior methods address either unimportant tokens or visual redundancy, while PACT tackles both by pruning irrelevant tokens and merging redundant ones.Source: §2.2 · p. 2

3. Method

• PACT applies EUTI and DBDPC to visual tokens at a selected language-model layer, then reintegrates initially unimportant tokens sufficiently close to cluster centers before merging.The reintegration threshold is α·dc; the method is described as applicable regardless of the visual encoder or connector.Source: §3 · p. 2, Figure 1 · p. 3, §3.2 · pp. 5–6
• PACT’s three stages are pruning unimportant tokens with EUTI, clustering the remaining tokens with DBDPC, and merging cluster members into a reduced visual-token set.Initially pruned tokens near cluster centers can be returned to those clusters before merging.Source: §3 · p. 2, Figure 1 · p. 3

3.1. Unimportant tokens identification

• EUTI combines hidden-state information with early-layer keys and queries to score tokens, then uses λ to set the percentage classified as unimportant.Its importance score uses a global query, head-wise key-query comparisons, and hidden-state norm; it avoids relying on attention scores, which may introduce positional bias.Source: §3.1 · pp. 3–4

3.2. Clustering-based merging of visual tokens

• DBDPC processes vectors by local density, selects centers separated by more than dc, and assigns each vector to its nearest center, guaranteeing distance below dc.This bound keeps cluster members relatively close, addressing the risk of distant points within large clusters.Source: §3.2 · pp. 4–5
• Merged hidden states receive their cluster center's position ID, while proportional attention weights visual tokens by cluster size and leaves textual-token weights at one.The reduction layer is chosen early for computational gain, but only where key vectors are sufficiently distinctive for pruning and clustering.Source: §3.2 · p. 6

4.1. Evaluation datasets

• PACT is evaluated on single-image, multi-image, video, document, reasoning, and visual-chat tasks.The datasets include benchmarks for text-rich documents, inter-image reasoning, video comprehension, and dialogue-based video interaction.Source: §4.1 · pp. 6–7

4.2. Evaluation setup

• Comparisons align methods by reduction ratio or average visual-token count, and include reduction-algorithm time in processing-time and throughput measurements.The experiments were conducted on a single A100 GPU; ToME uses a scheduler to reach reduction ratios above 50%.Source: §4.2 · p. 7

4.3. Results

• PACT consistently outperforms other methods at equal reduction ratios and equal throughput across four models.On LLaVA-OneVision-7B at a 98.6% Approach-to-Reference Metric Ratio, it achieves a 71.3% reduction ratio, 31% GPU memory reduction, and 225% speedup in language-model inference time.Source: §4.3 · p. 8
• DBDPC outperforms other clustering algorithms at equal reduction ratios and shows improved computational efficiency and throughput.The comparisons include agglomerative clustering, k-means, Density Peaks Clustering, and DBSCAN.Source: §4.3 · p. 8, Figure 9 · p. 8

4.4. Ablation study

• PACT outperforms pruning-only EUTI and clustering-only DBDPC across reduction ratios, supporting the benefit of combining token-irrelevance pruning with redundancy reduction.The ablations also report that each tested DBDPC component contributes positively; position-ID assignment is crucial, and proportional attention helps above 50% reduction.Source: §4.4 · pp. 8–9

5. Conclusion

• PACT addresses both visual-token irrelevance and redundancy without additional training.It is applied at inference time and does not rely on textual tokens for visual-token reduction.Source: §5 · p. 10
• PACT operates independently of the visual encoder and connector architecture, supporting application across visual language models.The conclusion characterizes it as a plug-and-play solution.Source: §5 · p. 10
• The paper concludes that visual language models use an unnecessarily large number of visual tokens and identifies efficient connectors and architectures as future work.Source: §5 · p. 10

A. On the density peaks clustering algorithm

• Density Peak Clustering selects centers using local density and distance to higher-density points, then assigns other points to the nearest higher-density point.Density can be measured by counting points within cutoff distance dc or by a Gaussian function; center selection can use a threshold on ρ_i · δ_i or a fixed percentage.Source: A. On the density peaks clustering… · p. 13

B. DBDPC Characteristics

• DBDPC selects a new center only when its minimum distance from existing centers exceeds dc, so centers are more than dc apart.Each point is assigned to its closest center; if that distance exceeded dc, the point would instead qualify as a new center.Source: B. DBDPC Characteristics · p. 13
• The supplementary proof states that DBDPC bounds intra-cluster distance by 2 dc (2 − dc).It derives the bound from each point being within dc of its center under distance 1 − cosine similarity.Source: B. DBDPC Characteristics · p. 13

C. A comparison between DBDPC and other clustering algorithms

• DBDPC keeps each element within dc of its center and bounds inter-cluster distances, avoiding DPC's unconstrained merges and overly close centers in dense regions.The stated bounds are below dc for element-to-center distance and 2dc × (2 −dc) for inter-cluster distance; center distances are at least dc.Source: C. A comparison between DBDPC and… · p. 13
• In Figure 12, DBDPC and DBSCAN identify isolated points, while DBDPC keeps cluster counts consistent across density differences; large clusters from DPC or DBSCAN may lose information when merged.The figure is illustrative, and the authors note DPC and DBSCAN behavior may vary with hyperparameters, while the discussed characteristics persist across choices.Source: C. A comparison between DBDPC and… · p. 14, Figure 12 · p. 14

D.1. A recursive approach

• The recursive method selects centers with δi > dc, excludes nearby vectors, and falls back to iterative processing when few new centers remain.This reduces sequential iterations and improves GPU parallelization while yielding the same clustering results; the authors report about 3× less computational time in practice.Source: §D.1 · pp. 14–15

D.2. Proof of correctness of the recursive approach

• The recursion preserves DBDPC center selection: selected centers satisfy the cutoff rule, excluded neighbors cannot become centers, and removed vectors cannot affect processing of the reduced set.The proof relies on selecting centers where δi > dc and excluding vectors within dc of those centers.Source: §D.2 · p. 15

E. On the choice of Positional IDs for clustering algorithms

• For k-means and agglomerative clustering, the representative is closest to the cluster average; for DBSCAN, the point connected to the most others is used, and each resulting vector receives the center's position ID.The DBSCAN choice performed slightly better in the authors' tests; they report each algorithm's best-performing merging or representative-token strategy.Source: E. On the choice of Positional IDs for… · pp. 15–16

F. More about applying ToME to Visual Language Models

• ToMe matches tokens across two sets by key-vector cosine similarity and merges the top ri pairs; a linearly decreasing schedule enables higher reduction ratios while reducing tokens at every layer.A fixed reduction ri = r cannot exceed 50% reduction unless later layers stop reducing tokens when their count falls below r.Source: F. More about applying ToME to Visual… · p. 16

G. Implementation details and hyperparameters for PACT

• For LLaVA-OneVision-7B, dn=2 and L=4 were set across experiments; α was 1.5 generally but 0.7 in Tab. 1, while reduction-ratio tests varied dc and λ.The authors chose dn and α by grid search on SeedBench [24], which they excluded from the testing datasets.Source: G. Implementation details and… · p. 16
• For LLaVA-1.6-Mistral-7B, Qwen2-VL-7B-Instruct, and InternVL2-8B, dn and α match LLaVA-OneVision; L=4 for Qwen2-VL and L=7 for the other two.The authors note that InternVL2-8B's reduction layer is less evident and choose L=7 without experimentally determining the optimum.Source: G. Implementation details and… · p. 16

H. More about test datasets and used metrics

• Evaluation uses LMMs-Eval and aims to follow [27], but unavailable closed-source versions led to GPT-4 assessments with gpt-4o-2024-08-06 and exclusion of API-scored datasets from reduction-ratio tests.The authors report API responses are not fully deterministic even at zero temperature; five high-inference-time datasets are also excluded from Tab. 1 throughput calculations.Source: H. More about test datasets and used… · p. 16
• Figure 13 illustrates maximum distances between visual-token keys across the first ten LLaVA-1.6-Mistral-7B layers before rotary embeddings.The passage identifies the plotted quantity and layer range but supplies no numerical values.Source: Figure 13 · p. 17
• Validation splits for DocVQA, InfoVQA, and TextVQA differ from [27], which may shift some LLaVA-OneVision results; comparisons remain consistent because all approaches use the same procedure.EgoSchema and PerceptionTest use submission-based test-set metrics.Source: H. More about test datasets and used… · pp. 16–17
• Results average three runs per dataset to account for slight variations from equal distances or importance metrics among vectors.The stated edge cases can produce variations when reduction methods are used.Source: H. More about test datasets and used… · p. 17
• Without reduction, Qwen2-VL-7B-Instruct had GPU out-of-memory errors on MLVU, VideoMME, and ActivityNet Perception, while ScienceQA scored 0.132; these datasets were excluded.The authors also note that LMM-Eval prompt formatting can make results differ from official results for some datasets.Source: H. More about test datasets and used… · p. 17

I. Additional numerical results

• DBDPC outperforms other clustering algorithms on most datasets and clusters significantly faster.The comparison uses an approximately 60% reduction ratio on LLaVA-OneVision-7B.Source: §I · p. 17
• Figures 14 and 15 show maximum distances between visual-token keys before rotary embeddings across the first 10 layers of Qwen2-VL-7B-Instruct and InternVL2-8B, respectively.Source: Figure 14 · p. 17, Figure 15 · p. 17
• EUTI outperforms FastV on most datasets and is more computationally efficient.The comparison uses an approximately 60% reduction ratio on LLaVA-OneVision-7B.Source: §I · p. 17
• Using keys for DBDPC distance calculations outperforms using hidden states on most test datasets.Source: §I · p. 17

J. Ablation study : Additional numerical results

• The PACT performance table reports results for LLaVA-OneVision-7B using dc = 0.17 and α = 0.7.Source: Table 5 · p. 20
• The comparisons cover PACT against FastV, VTW, and ToME on InternVL2-8B, and against FastV, Prumerge, and Hired on LLaVA-1.6-Mistral-7B.Source: Table 6 · p. 20, Table 7 · p. 20
• At approximately 60% reduction on LLaVA-OneVision-7B, the tables compare DBDPC with agglomerative clustering and with DBSCAN, DPC, and KMeans.Source: Table 8 · p. 21, Table 9 · p. 21
• An approximately 60% reduction comparison table covers EUTI-based visual-token pruning versus FastV on LLaVA-OneVision-7B.Source: Table 10 · p. 21
• At approximately 70% reduction, PACT outperforms standalone DBDPC and EUTI across datasets, while those methods show significant drops on some datasets.Source: J. Ablation study : Additional… · p. 18, Table 11 · p. 22
• The DBDPC ablation table examines center Position-IDs assignment; processing time is reported as similar across approaches.For the center Position-IDs ablation, hidden states are reordered by the mean Position-IDs of cluster elements before sequential IDs are assigned.Source: Table 12 · p. 22
• The EUTI pruning ablation and pruned-token recovery ablation report metrics rather than processing time, which is described as similar across the compared approaches.Removing token recovery is equivalent to setting α to zero.Source: Table 13 · p. 23, Table 14 · p. 23
• The keys-utilization ablation compares metrics for DBDPC at approximately 60% reduction, with processing time described as similar; experiment tables specify dataset splits and evaluation metrics, including exact-match accuracy where accuracy is reported.Source: Table 4 · p. 19, Table 15 · p. 23