PaperlayerView exact revision on arXiv

Source-linked AI summary

Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Driving

Minhao Xiong, Zichen Wen, Zhuangcheng Gu, Xuyang Liu, Rui Zhang, Hengrui Kang, Jiabing Yang, Junyuan Zhang, Weijia Li, Conghui He, Yafei Wang, Linfeng Zhang

arXiv:2508.13305v2cs.CV

TL;DR

Prune2Drive addresses the computational burden of high-resolution, multi-view visual inputs in autonomous-driving VLMs with diversity-aware token selection and view-adaptive pruning. At 10% token retention, it reports performance drops of 3% on DriveLM and 6% on DriveLMM-o1, using 13.4% and 20.3% of FLOPs, respectively.

  • Problem

    High-resolution multi-view images produce large visual-token sequences that increase VLM inference latency and memory use under quadratic attention complexity.

    Source: §1 · p. 2

  • Method

    Prune2Drive combines diversity-aware token selection across views with a controller that optimizes token-retention ratios for individual camera views.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    With 10% of tokens retained, Prune2Drive uses 13.4% and 20.3% of FLOPs, with performance drops of 3% and 6% on DriveLM and DriveLMM-o1, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported results support Prune2Drive as a plug-and-play token-pruning framework for multi-view VLMs in autonomous driving.

    Source: §5 · p. 8

    The theoretical analysis models prediction error as a weighted aggregation of per-view information degradation and assumes view-weighted Lipschitz continuity.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Vision-Language Models (VLMs) have emerged as a promising paradigm in autonomous driving (AD), providing a unified framework for perception and decision-making. However, their real-world deployment is hindered by significant computational overhead when processing high-resolution, multi-view images. This complexity stems from the massive number of visual tokens, which increases inference latency and memory consumption due to the quadratic complexity of self-attention. To address these challenges, we propose Prune2Drive, a plug-and-play visual token pruning framework for multi-view VLMs in AD. Prune2Drive introduces two core innovations: (i) a diversity-aware token selection mechanism that prioritizes semantic and spatial coverage across views, and (ii) a view-adaptive pruning controller that automatically learns optimal pruning ratios based on camera importance to downstream tasks. Unlike prior methods, Prune2Drive requires no model retraining or access to attention maps, ensuring compatibility with modern efficient attention implementations. Extensive experiments on the DriveLM and DriveLMM-o1 benchmarks demonstrate that Prune2Drive achieves significant speedups and memory savings with minimal performance impact. When retaining only 10% of visual tokens, our method achieves a 6.40x speedup in the prefilling phase and consumes only 13.4% of the original FLOPs, with a mere 3% average performance drop on the DriveLM benchmark. Code is available at: https://github.com/MinhaoXiong/Prune2Drive.git

1. Introduction

Multi-view VLMs face high inference latency and memory use from large visual-token sequences and quadratic attention. Prune2Drive combines diversity-aware token selection with per-view pruning and reports substantial efficiency gains with limited performance drops.

1. Introduction

  • High-resolution multi-view inputs increase visual-token counts, making inference slower and more memory-intensive under the quadratic attention complexity O(N^2).The described setup uses synchronized images from six surrounding cameras.Source: §1 · p. 2
  • Existing pruning methods often target single images or rely on attention weights, overlooking multi-view diversity and potentially excluding semantically important low-attention tokens.Attention-dependent strategies may be incompatible with efficient implementations such as Flash Attention.Source: §1 · p. 2
  • Prune2Drive combines T-FPS, which targets semantic and spatial coverage across views, with a controller that assigns distinct pruning ratios by camera importance.The controller is guided by cameras’ relative contributions to downstream driving tasks.Source: §1 · p. 2
  • The paper reports that Prune2Drive accelerates inference while maintaining, and sometimes improving, task performance on DriveLM and DriveLMM-o1.Figure 2 is described as showing state-of-the-art performance on both benchmarks.Source: §1 · p. 2, Figure 2 · p. 2
  • With 10% of tokens retained, Prune2Drive uses 13.4% and 20.3% of FLOPs, with performance drops of 3% and 6% on DriveLM and DriveLMM-o1, respectively.The reported results cover both autonomous-driving benchmarks.Source: §1 · p. 2

2. Related Work

Related work covers VLMs for autonomous driving and visual-token pruning for VLMs. It highlights deployment costs and gaps in existing pruning methods for multi-view settings.

Vision-Language Models for Autonomous Driving

  • VLM-based driving systems support language instructions and control, but multi-view inputs and large models contribute to latency and GPU-memory demands.These costs are identified as challenges for real-time deployment and safety in time-critical driving scenarios.Source: §2 · p. 3

Token Pruning for Vision-Language Models

  • Prior token-pruning methods include architectural additions that typically require retraining and training-free approaches that remove visual tokens during inference.Examples include attention-score selection, text-guided crossmodal pruning, and multi-stage selection.Source: §2 · p. 3
  • Most existing pruning methods rely on attention scores from a specific layer and focus on single-image inputs, leaving multi-view diversity and inter-view redundancy underexplored.The passage also notes positional bias and the risk of overlooking semantically important low-attention tokens.Source: §2 · p. 3

3. Methodology

Prune2Drive combines diversity-aware token selection with view-specific pruning-ratio optimization. Its theoretical analysis frames the resulting prediction error through view-weighted information loss.

3. Methodology

  • The framework pairs diversity-aware token pruning, which preserves non-redundant visual features, with per-view ratio optimization to allocate tokens by semantic importance.The combined design targets global perception completeness while preserving local features.Source: §3 · p. 3

3.1. Token-wise Farthest Point Sampling

  • T-FPS uses cosine distance in token-embedding space to iteratively select distant, non-redundant tokens and maximize semantic and spatial coverage.The method requires one pairwise-distance computation, taking 0.02s for 729 tokens per image and less than 0.1% of total FLOPs.Source: §3.1 · pp. 3–4

3.2. View-adaptive Pruning Ratio Optimization

  • The controller jointly optimizes per-camera retention ratios on a validation set, using a reward-penalty objective to balance task performance against total retained tokens.The search space can also impose a total token-budget constraint.Source: §3.2 · pp. 4–5

3.3. Theoretical Analysis

  • Under Assumption 1, view-adaptive diversity pruning achieves a tighter error bound than uniform ratios and random sampling at the same total token budget.The analysis attributes this to combining per-view token coverage with larger budgets for more important views.Source: §3.3 · p. 5

4. Experiments

Experiments compare Prune2Drive with token-pruning baselines across driving, regular VLM, and video-driving benchmarks, and examine efficiency, pruning choices, and selected tokens. Results report strong benchmark performance alongside substantial efficiency gains.

4.1. Experimental Settings

  • DriveMM and DriveLMM-o1 are evaluated on NuScenes-based DriveLM and DriveLMM-o1 VQA benchmarks against four training-free baselines at 75% and 90% average token pruning.The benchmarks include multiple driving tasks and use their respective common evaluation metrics.Source: §4.1 · pp. 5–6

4.2. Main Results

  • Prune2Drive outperforms other token-pruning methods on DriveLMM-o1 at both 75% and 90% average pruning, especially on risk assessment and scene awareness.The reported advantage is consistent across pruning ratios; high-risk-object and judgment performance strengthens under aggressive pruning.Source: Table 2 · p. 6, §4.2 · p. 6
  • With 75% of visual tokens pruned, Prune2Drive scores 58.3 on DriveLM, only 1% below the original model.The method also leads in average score and language metrics, and scores 34.0 on match score with 25% of visual tokens retained, 0.1 above the original model.Source: §4.2 · p. 6
  • With 128 tokens retained (22.2% of the original), Prune2Drive reaches 97.3% of full-token performance on regular VLM benchmarks.At 64 tokens, it outperforms SparseVLM by 7.5%.Source: §4.2 · pp. 6–7, Table 3 · p. 7
  • On OmniDrive, Prune2Drive scores 49.0 versus 50.3 for the vanilla model, exceeding FastV by 4.7 and SparseVLM by 2.2.The authors report effective capture of spatial-temporal features.Source: Table 5 · p. 7, §4.2 · p. 7
  • With 10% visual-token retention, the method achieves prefilling speedups of 6.40× on DriveMM and 2.64× on DriveLMM-o1, and decoding speedups of 1.09× and 1.04×, respectively.Relative to the original model, KV cache is 13.4% and FLOPs are 20.3%; experiments ran on a single NVIDIA A100-80GB GPU.Source: §4.2 · p. 7

4.3. Ablation study

  • TPE performs best for pruning-ratio optimization, while GridSearch and Evolutionary are slightly lower; cosine, ℓ1, and ℓ2 perform similarly, but nearest-first selection degrades performance.The authors interpret the nearest-first result as evidence that clustered tokens are redundant and do not capture visual diversity.Source: §4.3 · p. 7, Table 6 · p. 8, Table 7 · p. 8

4.4. Qualitative Analysis

  • Figure 4 contrasts FastV’s posterior-token duplicates and DART’s uniform sampling with Prune2Drive’s capture of critical cars in front and back-right views.The examples include two cars in the front view and one in the back-right view.Source: §4.4 · pp. 7–8, Figure 4 · p. 8

5. Conclusion

The conclusion presents Prune2Drive as a plug-and-play pruning framework for multi-view autonomous-driving VLMs. It combines diversity-aware token selection with view-adaptive pruning and reports strong performance across several benchmark types alongside reduced computational cost.

5. Conclusion

  • Prune2Drive is presented as the first plug-and-play visual-token pruning framework tailored to multi-view VLMs in autonomous driving.The conclusion frames the method around preserving semantic information and spatial coverage while optimizing pruning ratios for camera perspectives.Source: §5 · p. 8
  • Its two components are farthest-point-sampling-based diversity-aware token selection and a controller that dynamically optimizes pruning ratios for each camera view.The selection mechanism targets semantic information and spatial coverage among visual tokens.Source: §5 · p. 8
  • Experiments report state-of-the-art performance across VLM architectures and strong results on video and conventional VLM benchmarks, with lower computational costs and high task performance.Source: §5 · p. 8

A.1. Model architectures

DriveMM combines a SigLIP vision encoder, an MLP projector, and Llama-3.1 8B, trained through a four-stage curriculum. DriveLMM-o1 uses a fine-tuned InternVL2.5-8B model with dynamic image patching.

A.1.1. DriveMM

  • DriveMM uses SigLIP at 384×384 resolution, a 2-layer MLP projector, and Llama-3.1 8B.Its tokenizer has a 128,000-token vocabulary, and the model is trained on sequences of 8,192 tokens.Source: §A.1.1 · p. 11
  • DriveMM trains progressively across autonomous-driving datasets using a four-stage curriculum.The stages are Language-Image Alignment, Single-Image Pre-Training, Multi-Capacity Pre-Training, and Driving Fine-Tuning.Source: §A.1.1 · p. 11

A.1.2. DriveLMM-o1

  • DriveLMM-o1 is an InternVL2.5-8B model fine-tuned on its dataset, using dynamic image patching and LoRA.Training freezes the vision encoder and most language-model layers.Source: §A.1.2 · p. 11

A.2. Evaluation benchmarks

The appendix describes benchmark evaluation metrics for DriveLM and DriveLMM-o1 and presents a visual comparison of retained tokens. DriveLMM-o1 combines multiple reasoning-quality measures into an overall score.

A.2. Evaluation benchmarks

  • Evaluation metrics are adopted from the two benchmarks.Source: §A.2 · p. 11

A.2.1. DriveLM

  • DriveLM evaluation includes accuracy, BLEU, ROUGE-L, CIDEr, GPT-scored answers, bounding-box Match, and a weighted Average Score.The Average Score weights ChatGPT Score, Language Score, Match Score, and Accuracy at 0.4, 0.2, 0.2, and 0.2, respectively.Source: §A.2.1 · p. 11

A.2.2. DriveLMM-o1

  • DriveLMM-o1 evaluation uses GPT-4o-mini to assess reasoning and driving-response qualities, including risk, scene understanding, relevance, and missing information.Its evaluation description includes a 1–10 rubric for multiple reasoning metrics.Source: §A.2.2 · pp. 11–13
  • DriveLMM-o1’s overall score is the average of its metric scores, which evaluate reasoning quality across multiple dimensions.The listed measures include faithfulness, informativeness, risk assessment, traffic-rule adherence, hallucination, and missing details.Source: §A.2.2 · pp. 11–13

C.1. Comparison with baselines

  • Prune2Drive’s retained-token visualization shows view-importance-aware selection of informative tokens under aggressive pruning.The passage states that this consideration is crucial for downstream autonomous-driving tasks.Source: §C.1 · p. 13

C.2. Failure analysis

T-FPS can under-sample safety-critical objects when they occupy large, uniformly colored image regions.

C.2. Failure analysis

  • Uniform-color regions with similar image features may yield too few retained tokens for large safety-critical objects.The illustrated cases include an orange bus and three white vehicles that are sparsely represented.Source: §C.2 · p. 13

D.1. Evaluation Details

The appendix specifies evaluation procedures, optimization settings, and benchmark context, and reports comparable accuracy for similar λ values. It also identifies the lack of closed-loop evaluation as a limitation.

D.1. Evaluation Details

  • DriveLMM-o1 is evaluated on its full test set with GPT-4o-mini, while DriveLM results are submitted to the official leaderboard test server.Source: §D.1 · p. 13

D.2. Pruning ratio optimization settings

  • The pruning-ratio optimization settings are listed in Table 8.The supplied text does not specify the individual hyperparameter values.Source: §D.2 · p. 13, Table 8 · p. 13

D.3. Hyperparameter λ sensitivity.

D.4. Regular VLM benchmarks

  • The regular-VLM benchmark descriptions include GQA, MME, POPE, VQA V2, and VizWiz, covering scene reasoning, perception, hallucination, and visual question answering.VizWiz contains real-world images and questions from blind users.Source: §D.4 · p. 15

D.5. Video AD benchmark OmniDrive

  • OmniDrive is a nuScenes-derived multimodal autonomous-driving dataset with multi-view image and video inputs.It contains 374,329 training samples and 72,184 test samples and uses rule-based language metrics.Source: §D.5 · p. 15

E. Future Works

  • Experiments use offline autonomous-driving datasets, leaving performance in dynamic, interactive environments and long-term behavior unevaluated.The authors identify this as limiting understanding of real-world applicability.Source: E. Future Works · p. 15

Related papers

Loading 2508.13305v2…