Source-linked AI summary
VETO: Video Efficient Token Optimization for Vision Language Models
Gueter Josmy Faure, Hao Ping Wang, Min-Hung Chen, Winston H. Hsu
TL;DR
Long-video VLM inference faces quadratic visual-token attention costs, and single-axis compression has an efficiency floor. VETO applies spatial token merging before temporal frame merging, and the paper reports latency reductions across model scales alongside state-of-the-art performance at a 10% token budget.
Problem
Long videos generate many visual tokens, making VLM attention costly, while single-axis compression methods face an efficiency floor because they treat spatial and temporal redundancy independently.
Source: §1 · pp. 1–2Method
Results
Takeaways & Limitations
The reported results support dual-axis compression as an alternative to single-axis methods under severely constrained token budgets.
Source: §1 · p. 2Temporal compression can degrade non-linearly below Gratio = 0.50 because discarded frames may contain unique transitions or action states.
Source: §10 · p. 12
Abstract
from arXiv · showhide
Processing long videos with Vision-Language Models (VLMs) is bottlenecked by the quadratic cost of visual tokens, making long-form inference prohibitively expensive. While single-axis compression methods mitigate this, they hit a hard efficiency floor because they treat spatial and temporal redundancy independently. We present VETO (Video Efficient Token Optimization for Vision-Language Models), a training-optional plug-in that eliminates this bottleneck through dual-axis compression: (i) an intra-frame compressor that merges semantically similar tokens within each frame via optimal-transport inspired matching, and (ii) an inter-frame compressor that identifies and merges temporally redundant frames. The key design insight is hierarchical ordering: by first compressing spatial dimensions, VETO drastically reduces the cost of subsequent global temporal matching, bypassing the efficiency wall of single-axis approaches, with an advantage that grows with modern fully-fused attention infrastructure. Empirically, VETO achieves up to 45% faster inference (e.g., on LLaVA-OneVision-7B) while preserving or improving accuracy. Under extreme token starvation (10% budget), VETO outperforms VFlowOpt (54.9%), VisionZip (52.6%), and FastV (47.9%) with 55.7% accuracy. We demonstrate universal applicability across LLaVA-OneVision, InternVL-2.5, and LongVA, with zero-shot accuracy preserved or improved in all cases.
1. Introduction
Long-video inference is costly because visual-token attention grows quadratically, while existing single-axis compression methods face an efficiency floor. VETO combines spatial and temporal compression in a hierarchical, training-optional plug-in, with reported latency reductions and strong results at a 10% token budget.
1. Introduction
- Single-axis methods treat spatial and temporal redundancy independently, and temporal compression must compare fully uncompressed spatial tokens, creating a latency floor.Training-based methods require substantial compute, whereas training-free plug-ins are immediately deployable.Source: §1 · p. 2
- VETO first merges semantically similar tokens within frames, then merges temporally redundant frames using global matching on reduced features.The intra-frame step uses optimal-transport-inspired assignment; this ordering reduces the cost of temporal similarity calculations.Source: §1 · p. 2
- VETO's advantage over single-axis baselines grows on modern optimized attention infrastructure because fused kernels cannot absorb its total-sequence-length reduction.The paper also describes VETO as a drop-in module without custom CUDA kernels or architectural modifications.Source: §1 · p. 2
- 55.7% accuracy under a 10% token budget outperforms VFlowOpt (54.9%) and VisionZip (52.6%).The introduction characterizes VETO as state-of-the-art under this severely constrained budget.Source: §1 · p. 2
- VETO consistently reduces end-to-end latency while preserving or improving accuracy across LLaVA-OneVision, InternVL-2.5, and LongVA.The reported evaluation covers LLaVA-OneVision at 0.5B and 7B, InternVL-2.5 at 1B, and LongVA at 7B.Source: §1 · p. 2
2. Related Work
Prior VLM efficiency plug-ins reduce tokens through pruning, merging, or selection, but typically operate along one spatial or temporal axis. VETO distinguishes itself through ordered dual-axis compression.
2. Related Work
- VLM efficiency methods include pruning, merging, and adaptive selection, while video approaches also include spatial-temporal selection and hierarchical memory.The passage names FastV, VisionZip, VFlowOpt, ST-LLM, VideoLLM-MoD, and HERMES among these approaches.Source: §2 · p. 2
- Unlike prior works described as single-axis, VETO combines spatial and temporal compression in a strict hierarchical order.The passage frames this ordering as exploiting complementary spatial and temporal redundancy.Source: §2 · p. 2
3. VETO: Video Efficient Token Optimization
VETO compresses a video first within frames and then across frames, while keeping the feature dimensionality supplied to the frozen LLM unchanged. Its design reduces the spatial dimension before global temporal matching.
3.1. Motivation: The Complexity Bottleneck
- Standard attention scales as O((T · N)^2), while VETO's ordered compression changes the stated effective complexity to O(T · N^2/k + T^2 · C).The passage contrasts this with single-axis temporal methods, whose global matching uses fully uncompressed spatial tokens.Source: §3.1 · p. 3
3.2. Problem Formulation
- The formulation seeks intra- and inter-frame compression functions that reduce spatial and temporal dimensions while minimizing information loss measured through downstream task performance.The objective compares task performance on the original memory bank with performance after the two compression functions.Source: §3.2 · p. 3
3.3. Intra-Frame Compressor
- The Intra-Frame Compressor partitions tokens into source tokens and anchors, assigns each source to its most similar anchor, and size-weights merged features.Its spatial token count falls from N to approximately N/k; modulo-k partitioning keeps an anchor from every spatial region.Source: §3.3 · pp. 3–4
3.4. Inter-Frame Compressor
- The Inter-Frame Compressor identifies similar frames after intra-frame compression, so temporal similarity is computed on the reduced spatial dimension.It uses spatial mean-pooling to obtain frame representations.Source: §3.4 · p. 4, §3.4.1 · p. 4
3.4.1. Global Similarity Assessment
- Global similarity assessment forms a pairwise cosine-similarity matrix from frame representations and greedily selects top-m nonoverlapping pairs for merging.Pairs are merged by compression-size-weighted averaging, and iterations continue until the target frame count is reached.Source: §3.4.1 · p. 4
4. Experiments
Experiments evaluate VETO as a zero-shot plug-in across VLM families and benchmarks, then test supervised tasks, compression ablations, and qualitative behavior. Results show latency reductions with accuracy preserved or improved, while ablations favor intra-frame compression before temporal matching.
4.1. Implementation Details
- The zero-shot plug-in is inserted between the visual encoder and LLM, with end-to-end latency measured at batch size 1.Experiments use H100+FlashAttention-2 primarily, with V100 latency reported for InternVL-2.5-1B and LLaVA-OneVision-0.5B; zero-shot settings use k=2 and Gratio=0.75 for video benchmarks.Source: §4.1 · p. 4
4.2. Integration with Vision-Language Models
- On LongVA-7B, VETO reaches 53.5%/0.90× on VideoMME and 53.0%/0.86× on LongVideoBench, outperforming ToMe and HERMES on both benchmarks.On LLaVA-OneVision-7B, VETO matches Vanilla at 58.9% VideoMME accuracy with 15% lower latency (0.85×); on InternVL-2.5-1B, it reaches 48.1% versus 49.2% with 19% lower latency.Source: §4.2 · pp. 5–6
4.3. Fully Supervised Results
- VETO achieves 80.6% accuracy on MovieChat-1K, a 2.0% improvement over HERMES using the identical InstructBLIP (vicuna-7B) backbone.The visual encoder and LLM remain frozen in the fully supervised variant; only a lightweight linear adapter is optimized.Source: §4.3 · p. 6
- On BreakFast and COIN, VETO scores 94.4% and 92.7%, respectively, ranking second overall and remaining competitive with dedicated temporal architectures.The authors report that hierarchical token merging preserves fine-grained temporal structure for procedural understanding.Source: §4.3 · p. 6
4.4. Ablation Studies
- Intra-first is both fastest and most accurate: every alternative ordering is 7–16% slower, and Intra-first achieves the best accuracy on all three benchmarks.Spatial compression first reduces the feature dimensions used for temporal matching; alternatives compute similarities over fully uncompressed representations.Source: §4.4 · pp. 6–7
- VideoMME accuracy stays within 0.6% across spatial factors k=2–5, while temporal retention below 0.25 causes substantial degradation, including a drop from 58.9% to 52.1%.The authors recommend k=2 and Gratio≥0.75; spatial redundancy is more forgiving than temporally critical transitions.Source: §4.4 · p. 7
- At 10% token retention, VETO scores 55.7%, ahead of VFlowOpt (54.9%), VisionZip (52.6%), and FastV (47.9%).The authors attribute graceful degradation to both compressors operating at moderate individual ratios rather than forcing one axis to overcompensate.Source: §4.4 · p. 7
4.5. Qualitative Compression Visualizations
- Qualitative visualizations show inter-frame compression retaining scene-transition keyframes and intra-frame compression merging similar backgrounds while preserving distinct foreground objects.The inter-frame example reduces seven frames to three.Source: §4.5 · p. 7
5. Discussion
The discussion emphasizes that spatial redundancy is more safely compressed than temporal information, because transitions encode events important to reasoning. VETO exploits this asymmetry with aggressive spatial compression followed by more conservative temporal compression.
5. Discussion
- Figure 4 illustrates the asymmetry: the temporal stage retains scene-transition keyframes, while the spatial stage merges similar background patches and preserves distinct foreground objects.The illustrated frame selection reduces seven frames to three.Source: Figure 4 · p. 8, §5 · p. 8
- Video frames contain redundant textures and backgrounds, whereas temporal transitions carry causally important events and state changes; the authors recommend k=2 and Gratio≥0.75.Spatial accuracy remains within 0.6% across k=2–5, while temporal degradation becomes substantial below Gratio=0.25 and moderate at 0.50.Source: §5 · pp. 7–8
- On optimized attention infrastructure, VETO’s relative speedup margin over single-axis baselines grows because fused kernels absorb per-frame spatial reductions but not total-sequence-length reductions.The discussion recommends k=2 and Gratio=0.75 for latency-critical video QA, reporting a 15% latency reduction on H100+FA2 with no accuracy loss.Source: §5 · p. 8
6. Conclusion
VETO is a training-optional dual-axis compression plug-in for video VLMs, with reported latency reductions across model scales and strong results under extreme token budgets. The plug-in experiments cover models up to 7B, and the intra-frame compressor is content-based rather than query-aware.
6. Conclusion
- VETO reports latency ratios of 0.85–0.86× on LLaVA-OneVision-7B, 0.55× on LLaVA-OneVision-0.5B, and 0.86–0.90× on LongVA-7B.Under a 10% token budget, it outperforms single-axis methods; fully supervised BreakFast and COIN results are 94.4% and 92.7%.Source: §6 · p. 8
- Plug-in experiments cover models up to 7B, and the intra-frame compressor uses content-based rather than text-query-aware compression.Validation beyond 7B is left to future work; a query-conditioned extension could improve tasks requiring precise spatial localization.Source: §6 · p. 8
7. Complexity Analysis
The analysis quantifies vanilla attention’s quadratic token cost and VETO’s compressed pipeline, then shows how spatial and temporal reductions compound beyond either single axis.
7.1. Vanilla attention cost
- Vanilla self-attention over T frames with N tokens each costs O(T²N²) on the flattened T × N sequence.The section expresses the same complexity as O((T · N)^2).Source: §7.1 · p. 11
7.2. VETO end-to-end cost
- VETO’s dominant cost combines intra-frame similarity O(TN²/k), inter-frame matching O(T²C), and final attention O(G²T²N²/k²).The inter-frame similarity step mean-pools spatially compressed frames into C-dimensional vectors before computing a T × T cosine matrix.Source: §7.2 · p. 11
7.3. Speedup ratio
- The attention-speedup factor is (k/G)^2; at k = 2 and G = 0.75, attention FLOPs fall by ≈7.1× relative to vanilla.This ratio describes final attention cost, not total end-to-end latency.Source: §7.3 · p. 11
7.4. Concrete example at VideoMME settings
- At VideoMME settings, VETO’s compression overhead is ≈0.7% of vanilla attention, while net attention FLOPs fall by ≈7.1×.The example uses T = 64, N = 729, k = 2, and G = 0.75.Source: §7.4 · p. 11, Table 5 · p. 11
7.5. Single-axis methods cannot achieve the same reduction
- Intra-only yields a 4× attention-FLOP reduction and inter-only 1.8×, while VETO compounds the reductions to 7.1×.Inter-only also computes frame similarity on full-N vectors, adding overhead before token reduction.Source: §7.5 · p. 11
8. InternVL-2.5 on EgoSchema
On EgoSchema, VETO reduces latency on InternVL-2.5-1B with a small accuracy drop, and the passage reports latency gains across two distinct VLM families.
8. InternVL-2.5 on EgoSchema
- VETO achieves a 23% latency reduction with only a 0.7% accuracy drop on InternVL-2.5-1B on EgoSchema.Table 6 reports accuracy (%) and normalized latency, where lower latency is better.Source: §8 · p. 11, Table 6 · p. 11
- ToMe and HERMES are excluded from the InternVL-2.5 comparison because of limited support for that model family.Source: §8 · p. 11
- Together with LLaVA-OV-7B’s 60.2%/0.60× result, this shows latency gains across two architecturally distinct families on EgoSchema.The passage identifies the LLaVA-OV-7B result as coming from Table 1b of the main paper.Source: §8 · p. 11
9. Standard Action Recognition: UCF-101
The UCF-101 experiment tests VETO on short clips with less temporal redundancy than the paper’s long-form tasks, using a supervised frozen-backbone and linear-adapter setup.
9. Standard Action Recognition: UCF-101
- VETO reaches 94.4% on UCF-101, matching TSN at 94.2% and falling within 1.5% of TSM at 95.9%.The passage attributes the result to the uniform-anchor spatial matching preserving discriminative features at k = 2 compression.Source: §9 · p. 12, Table 7 · p. 12
- The experiment uses 100 frames per clip, k = 2, Gratio = 0.20, 20 epochs, and batch size 4.The aggressive temporal compression is described as reflecting short clips’ lower inter-frame redundancy.Source: §9 · p. 12
- The authors interpret the result as evidence that the dual-axis design generalizes to temporal scales beyond its primary long-video target.UCF-101 is characterized as a short-clip benchmark with less temporal redundancy than the main paper’s long-form tasks.Source: §9 · pp. 11–12
10. Extended Ablation Discussion
The extended ablations explain why operation order affects overhead, distinguish spatial from temporal compression sensitivity, and identify a recommended operating point.
10. Extended Ablation Discussion
- Although Intra-first and Inter-first reach the same final token count, Inter-first incurs 1.07×–1.09× overhead because matching uses full-N rather than N/k vectors.At the stated settings, the similarity vectors are 729-dimensional for Inter-first and 365-dimensional for Intra-first.Source: §10 · p. 12
- VideoMME accuracy degrades <0.6% across k ∈ {2, 3, 4, 5}, but drops 6.8% at Gratio = 0.125.The spatial settings retain 20–50% of per-frame tokens; Gratio = 0.125 retains 12.5% of frames.Source: §10 · p. 12
- Uniform modulo-k partitioning preserves spatial coverage by assigning every spatial region an anchor, supporting aggressive token merging.The passage contrasts this with content-adaptive anchor selection, which can leave peripheral areas without valid destinations.Source: §10 · p. 12
- Temporal compression is more sensitive because discarded frames can contain unique transitions or action states; degradation becomes nonlinear below Gratio = 0.50.Source: §10 · p. 12
- The recommended operating point is k = 2 and Gratio = 0.75, which the authors place in both axes’ empirically robust regime.Source: §10 · p. 12