Source-linked AI summary
Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models
Xiaoyang Guo, Guoping Luo, Jusheng Zhang, Keze Wang, Wenhao Wang
TL;DR
Video VLM adaptation is costly because frame sequences create many visual tokens, while compression can induce semantic drift. TBD uses teacher-guided distillation to adapt compressed students, consistently outperforming compression-only baselines, including 97.0% relative Vanilla accuracy for LLaVA-Video at R = 10%.
Problem
Video inputs create token explosions that make VLM fine-tuning and inference computationally prohibitive, while compressed-input tuning can shift precise visual attention toward ambiguous concepts.
Method
TBD freezes the pretrained backbone, trains LoRA adapters, compresses visual tokens, and distills full-token teacher semantics into a compressed student using multiple supervision signals.
Results
TBD consistently outperforms compression-only baselines across three video VLM backbones and four benchmarks under moderate and aggressive compression; on LLaVA-Video at R = 10%, it preserves 97.0% of Vanilla average accuracy.
Takeaways & Limitations
The findings frame efficient adaptation under token compression as a semantics preservation problem, with TBD improving the accuracy-efficiency trade-off especially under aggressive compression.
Takeaways & Limitations
Evaluation stresses retention ratios of R = 20% and R = 10%, including a setting where 90% of visual evidence is discarded.
Abstract
from arXiv · showhide
Adapting video vision-language models (VLMs) is computationally expensive because video inputs produce a large number of visual tokens, making both fine-tuning and inference costly. Although visual token compression can reduce this overhead, direct adaptation on compressed inputs often causes semantic drift and noticeable performance degradation. We present Token-Budget Distillation (TBD), a parameter-efficient fine-tuning framework for adapting video VLMs under a fixed token budget. TBD freezes the pretrained backbone, updates only LoRA adapters, and integrates FlashVID-based visual token compression into the video pathway. To preserve full-token semantics under compression, TBD employs a dual-path teacher-student design, where a full-token teacher provides stable supervision and a compressed student is optimized with task loss, answer-region KL distillation, GT-anchored margin distillation, and reliability-aware KD control. This design enables the student to recover the semantic behavior of the full-token model while remaining efficient under aggressive token reduction. We evaluate TBD on three video VLM backbones, including LLaVA-Video, LLaVA-OneVision, and Qwen3-VL-8B-Instruct, across four video understanding benchmarks. TBD consistently outperforms compression-only baselines under both moderate and aggressive compression. On LLaVA-Video at retention ratio R = 10 percent, TBD preserves 97.0 percent of the Vanilla model's average accuracy; on LLaVA-OneVision at R = 10 percent, it achieves an average score of 58.4 and matches 100.0 percent relative accuracy.
1 Introduction
Video VLM adaptation is costly because frame-rich inputs create token explosions, while compression can induce semantic drift. TBD addresses this by distilling full-token semantics into a compressed LoRA-adapted student and outperforms compression-only baselines across backbones and compression levels.
- Video inputs trigger a token explosion across frames, making both fine-tuning and inference computationally prohibitive.
- Direct adaptation on compressed visual tokens can shift attention from precise actions toward ambiguous concepts, degrading confidence and semantic reasoning.
- TBD freezes the pretrained backbone, updates LoRA adapters, and uses FlashVID compression within a dual-path teacher-student framework.
- Answer-region KL distillation, GT-anchored margin distillation, and dynamic KD warmup transfer full-token semantics to the compressed student.
- TBD consistently outperforms compression-only baselines across three backbones and four benchmarks under both moderate and aggressive compression.
- 97.0% of Vanilla average accuracy is preserved by TBD on LLaVA-Video at R = 10%, while LLaVA-OneVision reaches 100.0% relative accuracy at R = 10%.
2 Related Work
Related work spans video vision-language modeling, visual token compression, and parameter-efficient adaptation with knowledge distillation. Figure 1 situates TBD as a full-token teacher and compressed LoRA student under a fixed token budget.
- Video vision-language models: Video VLM research has progressed from conversational systems toward long-video memory, compact representations, temporal grounding, and efficient video understanding.
- Visual token compression for video VLMs: Visual token compression methods reduce computational burden through generic pruning or merging and video-specific modeling of temporal and spatiotemporal redundancy.
- Parameter-efficient adaptation and distillation: Parameter-efficient adaptation updates a small parameter subset, while knowledge distillation offers a complementary teacher-student route for efficient transfer.
- TBD combines a full-token teacher branch with a compressed LoRA student optimized under a fixed token budget.
3 Method
TBD adapts video VLMs under a strict token budget by training a compressed student against a full-token teacher while updating only LoRA adapters. Its reliability-aware objectives align answer-region predictions and teacher–student decision margins to preserve semantic reasoning after compression.
- 3.1 Problem Setup: Under a strict token budget, TBD aims to preserve full-token semantic reasoning while adapting the model efficiently to compressed visual representations.The setup constrains visual tokens through compression and optimizes the compressed branch with reliability-aware distillation.
- 3.2 Token-Budget Distillation Framework: TBD constructs dual teacher and student pathways from the same pretrained backbone, using full-token inputs for the teacher and compressed inputs for the student.The teacher disables compression and LoRA adapters, while the student is trainable; the teacher is discarded after adaptation.
- 3.3 Compression-Aware LoRA Adaptation: The framework freezes backbone parameters and updates only LoRA adapters while FlashVID compresses spatiotemporal tokens before they enter the language model.FlashVID remains training-free; gradients update the adapters through the compressed representations.
- 3.4 Reliable Distillation Under Token Compression: TBD applies answer-region KL distillation only where the teacher is correct and confident, filtering unreliable token-level supervision.The aligned answer horizon uses the shorter valid answer length, and the reliability mask selects positions exceeding a confidence threshold.
- 3.4 Reliable Distillation Under Token Compression: GT-anchored margin distillation uses Huber loss to preserve the teacher’s confidence gap between ground-truth and hardest-negative tokens.This targets margin collapse and transfers decisive boundary gaps rather than only softened output distributions.
- 3.4 Reliable Distillation Under Token Compression: The final objective combines autoregressive task supervision with KL and margin distillation, using warmup and capped dynamic weighting to limit early over-regularization.The task loss covers valid student answer positions, while the distillation weight is constrained relative to the cross-entropy loss.
4 Experiments
The experiments evaluate TBD across multiple video VLM backbones and challenging video-understanding benchmarks. Comparisons use common token-retention ratios against uncompressed and representative compression baselines.
- Training Data and Details: TBD is trained exclusively on the LLaVA-Video-178K video instruction-tuning dataset.The dataset provides diverse spatiotemporal annotations and multi-turn reasoning supervision.
- Evaluation Benchmarks: Evaluation covers VideoMME, EgoSchema, LongVideoBench, and MVBench, spanning short-to-long videos, egocentric reasoning, contextual memory, and multimodal understanding.These benchmarks are selected to test generalizability and robustness across distinct video-understanding demands.
- Model Backbones: The study instantiates TBD on LLaVA-Video, LLaVA-OneVision, and Qwen3-VL-8B-Instruct.The backbones differ in architectural paradigms, spatial-temporal pooling strategies, and parameter scales.
- Baselines and Compression Regimes: Comparisons include Vanilla, FastV, VisionZip, FastVID, and FlashVID across uncompressed, spatial, and spatiotemporal reduction settings.Vanilla serves as the theoretical upper bound, while the other methods represent training-free token-reduction baselines.
- Baselines and Compression Regimes: All compressed variants use identical visual-token retention ratios, with R = 20% for moderate compression and R = 10% for aggressive compression.The R = 10% setting discards 90% of visual evidence to stress-test semantic reasoning under severe compression.
4.2 Main Results
TBD improves the accuracy-efficiency trade-off of compressed video VLMs across three backbones and multiple retention ratios, while complementary distillation components strengthen performance under aggressive compression.
- Main results: TBD consistently improves the accuracy-efficiency trade-off over compression-only baselines across three video VLM backbones and different compression regimes.It achieves the strongest or near-strongest performance on most benchmarks under the same token budget.
- Ablation study: 58.6 average score results when answer-region KL distillation is added to the LoRA-only Student baseline, improving over its 58.3 score.The ablation measures LLaVA-Video at R = 10%.
- Ablation study: 58.8 average score follows the addition of GT-anchored margin distillation, indicating an additional gain beyond output-distribution matching.The margin term is described as preserving the teacher’s local decision boundary.
- Ablation study: 58.9 average score and 97.0% relative accuracy are achieved by the full TBD configuration on LLaVA-Video at R = 10%.The result comes from the ablation setting with the complete reliability filtering and CE-relative KD capping mechanisms.
- Ablation study: The complete gains arise from complementary distribution alignment, margin-based boundary preservation, and reliability-aware KD control.These mechanisms respectively align outputs, preserve discrimination, and stabilize noisy distillation signals.
4.4 Effect of Compression Modules
TBD remains effective with multiple compression front-ends, but final performance depends on compression-module quality, with FlashVID producing the strongest result in the comparison.
- Compression-module comparison: 58.9 average score and 97.0% relative accuracy make TBD (FlashVID) the strongest of four compression-module variants at R = 10%.TBD+FastVID, TBD+VisionZip, and TBD+FastV obtain 57.7, 57.1, and 55.0 average scores, respectively.
- Compression-module comparison: TBD remains effective across FastVID, VisionZip, and FastV, showing that the framework is not tied to one compression mechanism.All comparisons use LLaVA-Video at R = 10% with the same TBD training objective.
- Compression-module comparison: Compression quality still sets an upper bound on semantic preservation under the same token budget.The reported performance gap across front-ends indicates that reliable distillation does not eliminate the influence of compression quality.
4.5 Logit Distribution Alignment Analysis
The logit analysis compares compressed models with a full-token teacher on 100 VideoMME instances at R = 10%. TBD produces answer-choice distributions substantially closer to the teacher than compression-only inference.
- Setup: The analysis samples 100 VideoMME instances and compares teacher, compression-only baseline, and TBD student next-token logits.All compressed-model analyses use LLaVA-Video at R = 10%.
- Method: Per-sample KL divergence is computed after retaining and softmax-normalizing the A/B/C/D answer-choice logits.The comparison uses KL(P_T∥P_baseline) and KL(P_T∥P_distilled).
- Findings: The distilled student shows a substantially smaller mean KL divergence to the teacher than the compression-only baseline.The right panel presents kernel density estimates of the two KL distributions, while the left panel shows per-sample comparisons.
- Findings: TBD makes the distilled student substantially closer to the full-token teacher in answer-choice probability space.The analysis supports semantic recovery beyond benchmark-accuracy improvements.
4.6 Decision Boundary Analysis
TBD improves the compressed student's decision boundary under aggressive compression, producing stronger preference for correct answers than compression-only adaptation.
- 200 VideoMME instances were evaluated using the GT-anchored margin Δ = L_gt−L_max_wrong, where positive values indicate confident correct predictions.Negative margins indicate an incorrect preference.
- The distilled student's margin distribution shifts rightward, indicating a stronger preference for the correct answer.
- The distilled student achieves a larger mean margin and a lower negative-margin ratio than the compression-only baseline.
- The margin term explicitly sharpens local decision boundaries beyond KL distillation's overall distribution alignment.This explains why the full TBD objective outperforms weaker variants in Table 2.
4.7 Attention Drift Analysis
Attention analysis shows that TBD preserves the teacher's task-relevant visual focus in a student operating with 90% of visual tokens discarded.
- The full-token teacher exhibits sharp, localized attention focused on task-relevant foreground entities.These include the core action or specific object needed to answer the query.
- Despite discarding 90% of the original tokens, the TBD-adapted student's attention map remains highly consistent with the teacher's.
4.8 Discussion
TBD treats compressed adaptation as teacher-guided semantic recovery, with complementary distillation mechanisms supporting performance as compression becomes more aggressive.
- TBD reframes token-compressed adaptation as teacher-guided semantics recovery rather than plain task fitting.
- Answer-region KL aligns output distributions, GT-anchored margin distillation preserves discrimination, and reliability-aware filtering stabilizes noisy signals.
- TBD generalizes across compression front-ends, while its upper bound is set by front-end quality.
5 Conclusion and Limitations
TBD combines token compression, LoRA, and reliable distillation to recover full-token semantic behavior and improve the accuracy-efficiency trade-off, especially under aggressive compression.
- TBD is a parameter-efficient framework for adapting video VLMs under strict token budgets.
- TBD consistently improves the accuracy-efficiency trade-off over compression-only baselines across multiple benchmarks and three representative backbones.The improvement is especially pronounced under aggressive compression.
- TBD requires a full-token teacher during adaptation, maintaining high training-time memory demands despite inference efficiency.
- The student's performance upper bound is constrained by irreversible information loss from the chosen training-free compression module.