Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models

• Video inputs trigger a token explosion across frames, making both fine-tuning and inference computationally prohibitive.Source: §1 · p. 1
• Direct adaptation on compressed visual tokens can shift attention from precise actions toward ambiguous concepts, degrading confidence and semantic reasoning.Source: §1 · p. 2
• TBD freezes the pretrained backbone, updates LoRA adapters, and uses FlashVID compression within a dual-path teacher-student framework.Source: §1 · p. 2
• Answer-region KL distillation, GT-anchored margin distillation, and dynamic KD warmup transfer full-token semantics to the compressed student.Source: §1 · p. 2
• TBD consistently outperforms compression-only baselines across three backbones and four benchmarks under both moderate and aggressive compression.Source: §1 · p. 2
• 97.0% of Vanilla average accuracy is preserved by TBD on LLaVA-Video at R = 10%, while LLaVA-OneVision reaches 100.0% relative accuracy at R = 10%.Source: §1 · p. 2

Video vision-language models

• Video VLM research has progressed from conversational systems toward long-video memory, compact representations, temporal grounding, and efficient video understanding.Source: §2 · p. 2

Visual token compression for video VLMs

• Visual token compression methods reduce computational burden through generic pruning or merging and video-specific modeling of temporal and spatiotemporal redundancy.Source: §2 · p. 2

Parameter-efficient adaptation and distillation

• Parameter-efficient adaptation updates a small parameter subset, while knowledge distillation offers a complementary teacher-student route for efficient transfer.Source: §2 · p. 2

• TBD combines a full-token teacher branch with a compressed LoRA student optimized under a fixed token budget.Source: Figure 1 · p. 3

3.1 Problem Setup

• Under a strict token budget, TBD aims to preserve full-token semantic reasoning while adapting the model efficiently to compressed visual representations.The setup constrains visual tokens through compression and optimizes the compressed branch with reliability-aware distillation.Source: §3.1 · p. 3

3.2 Token-Budget Distillation Framework

• TBD constructs dual teacher and student pathways from the same pretrained backbone, using full-token inputs for the teacher and compressed inputs for the student.The teacher disables compression and LoRA adapters, while the student is trainable; the teacher is discarded after adaptation.Source: §3.2 · pp. 3–4

3.3 Compression-Aware LoRA Adaptation

• The framework freezes backbone parameters and updates only LoRA adapters while FlashVID compresses spatiotemporal tokens before they enter the language model.FlashVID remains training-free; gradients update the adapters through the compressed representations.Source: §3.3 · p. 4

3.4 Reliable Distillation Under Token Compression

• TBD applies answer-region KL distillation only where the teacher is correct and confident, filtering unreliable token-level supervision.The aligned answer horizon uses the shorter valid answer length, and the reliability mask selects positions exceeding a confidence threshold.Source: §3.4 · p. 4
• GT-anchored margin distillation uses Huber loss to preserve the teacher’s confidence gap between ground-truth and hardest-negative tokens.This targets margin collapse and transfers decisive boundary gaps rather than only softened output distributions.Source: §3.4 · p. 4
• The final objective combines autoregressive task supervision with KL and margin distillation, using warmup and capped dynamic weighting to limit early over-regularization.The task loss covers valid student answer positions, while the distillation weight is constrained relative to the cross-entropy loss.Source: §3.5 · pp. 4–5

Training Data and Details

• TBD is trained exclusively on the LLaVA-Video-178K video instruction-tuning dataset.The dataset provides diverse spatiotemporal annotations and multi-turn reasoning supervision.Source: §4.1 · p. 5

Evaluation Benchmarks

• Evaluation covers VideoMME, EgoSchema, LongVideoBench, and MVBench, spanning short-to-long videos, egocentric reasoning, contextual memory, and multimodal understanding.These benchmarks are selected to test generalizability and robustness across distinct video-understanding demands.Source: §4.1 · p. 5

Model Backbones

• The study instantiates TBD on LLaVA-Video, LLaVA-OneVision, and Qwen3-VL-8B-Instruct.The backbones differ in architectural paradigms, spatial-temporal pooling strategies, and parameter scales.Source: §4.1 · p. 5

Baselines and Compression Regimes

• Comparisons include Vanilla, FastV, VisionZip, FastVID, and FlashVID across uncompressed, spatial, and spatiotemporal reduction settings.Vanilla serves as the theoretical upper bound, while the other methods represent training-free token-reduction baselines.Source: §4.1 · p. 5
• All compressed variants use identical visual-token retention ratios, with R = 20% for moderate compression and R = 10% for aggressive compression.The R = 10% setting discards 90% of visual evidence to stress-test semantic reasoning under severe compression.Source: §4.1 · p. 5

Main results

• TBD consistently improves the accuracy-efficiency trade-off over compression-only baselines across three video VLM backbones and different compression regimes.It achieves the strongest or near-strongest performance on most benchmarks under the same token budget.Source: §4.2 · p. 5

Ablation study

• 58.6 average score results when answer-region KL distillation is added to the LoRA-only Student baseline, improving over its 58.3 score.The ablation measures LLaVA-Video at R = 10%.Source: §4.3 · pp. 5–6
• 58.8 average score follows the addition of GT-anchored margin distillation, indicating an additional gain beyond output-distribution matching.The margin term is described as preserving the teacher’s local decision boundary.Source: §4.3 · pp. 6–7
• 58.9 average score and 97.0% relative accuracy are achieved by the full TBD configuration on LLaVA-Video at R = 10%.The result comes from the ablation setting with the complete reliability filtering and CE-relative KD capping mechanisms.Source: §4.3 · p. 6
• The complete gains arise from complementary distribution alignment, margin-based boundary preservation, and reliability-aware KD control.These mechanisms respectively align outputs, preserve discrimination, and stabilize noisy distillation signals.Source: §4.3 · pp. 6–7

Compression-module comparison

• 58.9 average score and 97.0% relative accuracy make TBD (FlashVID) the strongest of four compression-module variants at R = 10%.TBD+FastVID, TBD+VisionZip, and TBD+FastV obtain 57.7, 57.1, and 55.0 average scores, respectively.Source: §4.4 · p. 7
• TBD remains effective across FastVID, VisionZip, and FastV, showing that the framework is not tied to one compression mechanism.All comparisons use LLaVA-Video at R = 10% with the same TBD training objective.Source: §4.4 · p. 7
• Compression quality still sets an upper bound on semantic preservation under the same token budget.The reported performance gap across front-ends indicates that reliable distillation does not eliminate the influence of compression quality.Source: §4.4 · p. 7

Setup

• The analysis samples 100 VideoMME instances and compares teacher, compression-only baseline, and TBD student next-token logits.All compressed-model analyses use LLaVA-Video at R = 10%.Source: §4.5 · p. 7

Method

• Per-sample KL divergence is computed after retaining and softmax-normalizing the A/B/C/D answer-choice logits.The comparison uses KL(P_T∥P_baseline) and KL(P_T∥P_distilled).Source: §4.5 · p. 7

Findings

• The distilled student shows a substantially smaller mean KL divergence to the teacher than the compression-only baseline.The right panel presents kernel density estimates of the two KL distributions, while the left panel shows per-sample comparisons.Source: Figure 2 · p. 7
• TBD makes the distilled student substantially closer to the full-token teacher in answer-choice probability space.The analysis supports semantic recovery beyond benchmark-accuracy improvements.Source: §4.5 · p. 7, Figure 2 · p. 7

• 200 VideoMME instances were evaluated using the GT-anchored margin Δ = L_gt−L_max_wrong, where positive values indicate confident correct predictions.Negative margins indicate an incorrect preference.Source: §4.6 · p. 7
• The distilled student's margin distribution shifts rightward, indicating a stronger preference for the correct answer.Source: §4.6 · p. 7
• The distilled student achieves a larger mean margin and a lower negative-margin ratio than the compression-only baseline.Source: Figure 3 · p. 8
• The margin term explicitly sharpens local decision boundaries beyond KL distillation's overall distribution alignment.This explains why the full TBD objective outperforms weaker variants in Table 2.Source: §4.6 · p. 8

• The full-token teacher exhibits sharp, localized attention focused on task-relevant foreground entities.These include the core action or specific object needed to answer the query.Source: §4.7 · p. 8
• Despite discarding 90% of the original tokens, the TBD-adapted student's attention map remains highly consistent with the teacher's.Source: §4.7 · p. 8

• TBD reframes token-compressed adaptation as teacher-guided semantics recovery rather than plain task fitting.Source: §4.8 · p. 8
• Answer-region KL aligns output distributions, GT-anchored margin distillation preserves discrimination, and reliability-aware filtering stabilizes noisy signals.Source: §4.8 · p. 8
• TBD generalizes across compression front-ends, while its upper bound is set by front-end quality.Source: §4.8 · p. 8

• TBD is a parameter-efficient framework for adapting video VLMs under strict token budgets.Source: §5 · p. 8
• TBD consistently improves the accuracy-efficiency trade-off over compression-only baselines across multiple benchmarks and three representative backbones.The improvement is especially pronounced under aggressive compression.Source: §5 · p. 8
• TBD requires a full-token teacher during adaptation, maintaining high training-time memory demands despite inference efficiency.Source: §5 · p. 8
• The student's performance upper bound is constrained by irreversible information loss from the chosen training-free compression module.Source: §5 · p. 8