Source-linked AI summary
ContextTom: Training-Free Video VLM Acceleration with Early Spatial Context
Yuqi Li, Neil Rodrigues, Xianglong Wang, Junming Liu, Yi Xiao, Weilun Feng, Chuangguang Yang, Yingli Tian, Zhongwei Cheng, Wei Li, Lin Chen
TL;DR
Long video inputs make vision encoding and language-model prefill costly, while token reduction also determines how much computation and visual evidence precede removal. ContextTom compresses time before vision encoding and delays spatial selection until after a short multimodal prefix. Across the reported evaluations it reduces TTFT while maintaining or improving accuracy in several settings; on Video-MME short, Qwen3-VL-8B reaches 76.22% accuracy at 123.5 ms.
Problem
Long visual sequences make vision encoding and language-model prefill costly, and the timing of token removal determines how much computation tokens receive before removal.
Source: §1 · p. 1Method
ContextTom uses temporal medoid anchors before vision encoding, preserves their full spatial grids through a short decoder prefix, then selects query-relevant spatial representatives and continues from retained states.
Source: §1 · p. 2Results
Across reported Qwen settings, ContextTom has the lowest TTFT among compared methods; on Video-MME short, it achieves 76.22% accuracy at 123.5 ms with a 2.77× speedup over full-input inference.
Source: §1 · p. 2, §4.1 · p. 7Takeaways & Limitations
The reported findings favor allocating computation by stage: reduce temporal repetition before vision encoding, preserve full spatial access during early multimodal processing, and select representatives for later layers.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Video vision-language models process thousands of visual tokens before producing an answer. Reducing these tokens early saves computation, but also determines which evidence reaches the language model. We introduce ContextTom, a training-free framework that separates the timing of temporal and spatial compression. Its central idea is to compress time before vision encoding while preserving spatial coverage through early multimodal processing. A fixed-budget temporal partition produces representative medoid anchors, supplemented with bounded local detail from neighboring observations. The anchors retain their complete spatial grids through a short decoder prefix, allowing question tokens to access the visual evidence and completing Qwen3-VL's multi-level visual injections. ContextTom then selects spatial representatives using query attention and agreement across visual feature levels, and continues directly from their computed states. This schedule reduces vision computation and later decoder workload while reusing the prefix. On Qwen3-VL-8B, ContextTom achieves 76.22% accuracy on Video-MME Short with a mean time-to-first-token (TTFT) of 123.5 ms, yielding a 2.77x speedup over full-input inference. It matches the full-input accuracy of 72.20% on EgoSchema with a 2.82x speedup and achieves a 5.54x speedup on LongVideoBench at 59.24% accuracy.
1 INTRODUCTION
ContextTom treats the timing of token removal as a design choice: it compresses temporal observations before vision encoding but delays spatial selection until early multimodal processing has exposed visual evidence to question tokens. It then continues from the retained decoder states, aiming to reduce vision and later-decoder computation.
1 INTRODUCTION
- The design question is not only how many visual tokens survive, but how much computation they receive before removal.In Qwen3-VL, pruning spatial tokens before early decoder layers limits visual-injection locations and evidence accessible to question tokens, whereas keeping all tokens through the decoder burdens later layers.Source: §1 · pp. 1–2
- Prior token-compression methods merge or select visual features, use early decoder attention, or exploit repetition across video frames; these approaches motivate treating the compression boundary as part of the algorithm.The introduction specifically identifies ToMe, VisionZip, FastV, and EarlyTom as examples.Source: §1 · p. 1
- ContextTom allocates temporal and spatial compression to different stages: medoid anchors with bounded local detail reduce observations before vision encoding, and spatial representatives are selected after a short decoder prefix.Fixed-budget change-and-drift segmentation sets the intervals; the prefix retains each anchor's full grid while question tokens attend to it.Source: §1 · p. 2
- The framework targets both major inference costs by reducing grids entering vision blocks and shortening the sequence processed by later decoder layers.Retained states continue directly from the prefix, reusing the computation that guides spatial selection.Source: §1 · p. 2
- 76.22% accuracy at 123.5 ms mean TTFT on Video-MME short yields a 2.77× speedup over full-input inference for Qwen3-VL-8B.The 900-question subset result accompanies a reported 0.60-point accuracy gain from preserving the three-block full-spatial prefix over pre-decoder spatial compression, for 6.1 ms additional TTFT.Source: §1 · p. 2
2 RELATED WORK
Related work addresses visual-token redundancy, frame redundancy, and early compression at different points in the inference pipeline. ContextTom combines temporal reduction before vision encoding with spatial selection after Qwen3-VL's visual-injection prefix.
Visual token compression
- Visual-token methods merge similar tokens, select and consolidate features before language processing, or prune using early decoder attention (Bolya et al., 2023; Yang et al., 2025).AIM combines iterative similarity-based merging with progressive pruning, while PriorTR separates task-conditioned information from model-induced attention priors (Chen et al., 2026).Source: §2 · p. 2
Video redundancy
- Video-redundancy approaches select informative frames using feature-trajectory residuals or question-agnostic clustering, then may select visual tokens by importance and redundancy (Kim et al., 2026; Song et al., 2026).The passage also identifies temporal segmentation with density-based pruning as an approach (FastVID).Source: §2 · p. 2
- Other video methods combine temporal and spatial reduction outside the language model with merging or pruning inside it, or vary compression across frames (Shen et al., 2025; Shao et al., 2025; Fu et al., 2025; Liu et al., 2025).ContextTom instead uses pixel descriptors for temporal anchors before vision blocks, then query importance and multi-level visual similarity after the visual-injection prefix, with a shared spatial budget across frames.Source: §2 · p. 3
Early compression and visual injection
- ContextTom places fixed-budget medoid anchors before the first vision block and moves spatial selection to the end of Qwen3-VL's DeepStack injection prefix, with direct continuation from selected states.This is contrasted with EarlyTom, which merges frames within the vision encoder and applies spatial selection policies to dynamic and static frames.Source: §2 · p. 3
3 METHOD
ContextTom compresses time before vision encoding, preserves full spatial grids through a short multimodal decoder prefix, and then selects spatial representatives for later layers. It continues from retained prefix states while preserving text tokens and positions.
3.1 OVERVIEW: COMPRESS TIME EARLY, PRESERVE SPACE THROUGH THE PREFIX
- The overview retains K temporal anchors, processes their full spatial grids through the first p decoder blocks, then reduces the sequence for remaining blocks while keeping all nonvisual tokens.The Qwen3-VL-8B configuration uses 36 decoder blocks, p = 3, temporal retention rt = 0.52, spatial target rs = 0.85, and per-frame floor f = 0.80.Source: §3.1 · p. 4
3.2 TEMPORAL ANCHORS WITH LOCAL DETAIL
- Temporal intervals are greedily split using adjacent change and drift from the interval start until exactly K intervals remain, with each interval represented by its medoid.Nearby units in an interval can supplement corresponding anchor cells: only the top ⌊ρS⌋ cells receive bounded donor contributions, and fusion occurs before the first vision block so all vision blocks process K grids.Source: §3.2 · p. 4
3.3 PRESERVE SPATIAL ACCESS THROUGH VISUAL INJECTION
- All anchor cells, original text and timestamps, and multimodal position IDs pass through the first p decoder blocks with DeepStack additions before spatial selection.The resulting hidden states provide query-importance scores and starting states for later decoder blocks; question tokens can access complete retained-anchor grids through the prefix.Source: §3.3 · pp. 4–5
3.4 SELECT SPATIAL REPRESENTATIVES UNDER COVERAGE CONSTRAINTS
- Spatial candidates pair adjacent cells and combine minimum cosine similarity across visual feature levels with query-importance ranks; larger scores favor reduction while protecting pairs whose members both matter.A video-wide budget allocates removals across frames subject to per-frame retention floors, and each selected pair keeps its higher-importance cell gathered from the computed prefix states.Source: §3.4 · p. 5
3.5 CONTINUE FROM THE RETAINED STATES
- The retained visual states and all text tokens continue through later decoder blocks with their original positions, reusing prefix computation; early layers keep full-prefix KV histories while later layers cache the reduced sequence.The schedule reduces grids processed by vision blocks and sequence length in L − p decoder blocks; implementation targets single-request SDPA inference and measures model-side TTFT through first-token logits after video preprocessing.Source: §3.5 · p. 5
4 EXPERIMENTS
Experiments assess ContextTom’s accuracy–latency trade-off across video benchmarks, the sources of its efficiency gains, and the effects of spatial-compression timing, components, video length, and temporal retention. Results show speedups across benchmarks and backbones, with analyses of the associated accuracy and inference-cost trade-offs.
4 EXPERIMENTS
- Experiments cover Video-MME, EgoSchema, LongVideoBench, and MVBench across four backbones, using 1 FPS inputs for Qwen and 32 frames for InternVL and LLaVA.They report accuracy, mean TTFT, and speedup relative to full-input TTFT; component and timing analyses use Qwen3-VL-8B on Video-MME short.Source: §4 · p. 6
4.1 RQ1: ACCURACY AND LATENCY ACROSS VIDEO BENCHMARKS
- 76.22% accuracy at 123.5 ms TTFT on Video-MME short gives Qwen3-VL-8B a 2.77× speedup over full input; it matches full-input accuracy on EgoSchema and reaches 5.54× speedup on LongVideoBench.On LongVideoBench, accuracy is 59.24% at 157.5 ms, a 2.84-point decrease; on EgoSchema, accuracy is 72.20%.Source: §4.1 · p. 7
- On Video-MME overall, ContextTom exceeds full-input accuracy by 0.30 and 0.11 percentage points with 2.39× and 1.74× TTFT speedups on InternVL3.5-8B and LLaVA-OneVision-7B, respectively.It has the highest accuracy among compressed methods on medium, long, and overall subsets for both backbones; the reported table uses 32 input frames.Source: §4.1 · p. 7, Table 2 · p. 7
4.2 RQ2: SOURCES OF EFFICIENCY GAINS
- Decoder prefill provides the largest absolute saving at 90.56 ms, followed by vision encoding at 45.84 ms; together they account for 99.1% of measured TTFT reduction.The development comparison also reduces TTFT by 22.56% and end-to-end latency by 9.97%; preprocessing stays near 750 ms.Source: §4.2 · pp. 7–8, Table 3 · p. 8
4.3 RQ3: VALUE OF EARLY SPATIAL CONTEXT
- Delaying spatial selection until after three decoder blocks improves accuracy by 0.60 points for 6.1 ms additional TTFT, with a 0.35 GB increase in peak GPU memory.Only three of 36 decoder blocks process complete anchor grids.Source: §4.3 · p. 8, Table 4 · p. 8
4.4 RQ4: CONTRIBUTIONS OF TEMPORAL AND SPATIAL PROCESSING
- Removing local supplementation costs 1.66 accuracy points, while uniform temporal selection costs 0.90 points; bounded supplementation enriches anchors without adding vision-transformer grids.The Table 5 component ablations use the Table 1 configuration as their common reference.Source: §4.4 · pp. 8–9, Table 5 · p. 9
- Removing query information causes the largest ablation drop, 2.44 points, while using only final visual features costs 1.78 points; ContextTom also gains 0.44 points and lowers TTFT 5.51% versus temporal-only processing.The similarity decision uses agreement across final and intermediate visual features.Source: §4.4 · p. 9, Table 5 · p. 9
4.5 RQ5: VIDEO LENGTH AND TEMPORAL RETENTION BUDGET
- Across Video-MME short, medium, and long subsets, TTFT speedups are 2.77×, 2.87×, and 2.82×, while absolute savings rise from approximately 219 to 571 and 685 ms.Increasing temporal retention from 0.40 to 0.52 gains 0.68 accuracy points for 6.7 ms; increasing it next to 0.65 gains 0.25 points for 9.7 ms.Source: §4.5 · p. 9, Table 6 · p. 9
5 CONCLUSION
ContextTom’s computation schedule achieves the lowest TTFT among compared methods across the reported Qwen settings. The conclusion favors representative temporal inputs, an intact full-spatial prefix, and query-guided later selection with direct state reuse.
5 CONCLUSION
- The schedule achieves the lowest TTFT among compared methods across the reported Qwen settings.Source: §5 · p. 10
- Compression-timing analysis supports preserving complete spatial access during a short decoder prefix, while temporal-budget analysis shows diminishing accuracy returns from additional observations.Source: §5 · p. 10
- The design removes temporal repetition before vision encoding, then carries selected states directly into a shorter decoder suffix while reusing selection computation.Source: §5 · p. 10
AI USE STATEMENT
The statement identifies ChatGPT’s manuscript-assistance roles and the authors’ responsibility for the final work. The descriptor and anchor procedure uses compact RGB summaries for segmentation and representative selection.
AI USE STATEMENT
- The authors report ChatGPT assistance with organization, language polishing, LaTeX formatting, interpreting supplied experimental results, and formulating a property and proof; authors retain responsibility for the final manuscript.Source: AI USE STATEMENT · p. 11
A.1 DESCRIPTORS, SEGMENTATION, AND ANCHORS
- Each temporal unit is summarized by pooled per-cell RGB means, flattened into compact descriptors used for CPU-side segmentation and anchor selection.The spatial map is adaptively pooled to at most 8 × 8 cells.Source: §A.1 · p. 13
- Greedy segmentation recomputes interval candidates after each cut, applies deterministic tie-breaking, and emits anchors in temporal order.The implementation streams distances rather than storing a dense descriptor-distance tensor, while segmentation incurs interval rescans.Source: §A.1 · p. 13
A.2 DETAIL AND NOVELTY WEIGHTS
The appendix specifies how nearby observations supplement anchor cells, how the decoder prefix and query-dependent scores define the spatial-selection boundary, and how selected states continue through later layers.
A.2 DETAIL AND NOVELTY WEIGHTS
- Nearby same-interval units donate to corresponding anchor cells through detail-and-novelty scoring with bounded contributions; motion, noise, and occlusion may influence donor selection.Source: §A.2 · p. 13
B.1 DEEPSTACK BOUNDARY
- The prefix inserts DeepStack features at visual positions, with selection after the final injection; visual positions precede the question, while question states can access visual evidence and guide selection.The stated configuration uses J = p = 3, placing scoring after the last injection but before another complete decoder block.Source: §B.1 · p. 13
B.2 CAUSALLY NORMALIZED IMPORTANCE
- Importance scoring uses post-video text queries against visual keys, with all causally visible text and visual keys in the denominator.The query is computed only for the suffix, keys cover the full prefix, and the extra score-product cost is approximately O(nh|Q|n0dh).Source: §B.2 · p. 14
C.1 PAIR CONSTRUCTION AND DETERMINISTIC SELECTION
- Disjoint within-frame neighboring pairs are ranked using query importance and multi-level similarity; per-frame and global caps govern selection, retaining the higher-importance member of each chosen pair.Equal-importance ties favor the cell with the larger squared norm of its final visual embedding.Source: §C.1 · p. 14
C.2 POSITIONS AND CACHES
- Prompt assembly preserves original positions, and spatial gathering keeps surviving position IDs; early layers retain the full prefix KV history while later layers store the reduced sequence.Generated tokens are appended to each layer’s own history.Source: §C.2 · p. 14
C.3 COMPLETE INFERENCE ALGORITHM
- The inference procedure builds temporal anchors, encodes full anchor grids, runs the complete prefix, selects spatial pairs, gathers retained states, and continues through the decoder suffix.It keeps all text and generates using full early-layer and reduced later-layer caches.Source: §C.3 · p. 14
D.1 COST AND IMPLEMENTATION
The cost discussion separates decoder computation from temporal, scoring, and selection overhead, and notes implementation choices for single-request inference.
D.1 COST AND IMPLEMENTATION
- The approximate cost decomposition adds temporal, scoring, and selection costs to decoder-layer computation.Source: §D.1 · p. 15
- Decoder computation includes terms linear in sequence length and attention interactions quadratic in length, while the early prefix is paid once.Source: §D.1 · p. 15
- The implementation batches equal-length independent vision-frame attention and uses fused normalization and rotary kernels with compact sequence assembly.The main configuration uses dense temporal fusion and greedy segmentation for single-request SDPA inference.Source: §D.1 · p. 15
D.2 DEVELOPMENT CONFIGURATION AND TIMING
Development profiling reports model-side TTFT across temporal selection, visual processing, and decoder execution. It also specifies the synchronized timing interval and warmup procedure.
D.2 DEVELOPMENT CONFIGURATION AND TIMING
- Model-side TTFT runs from before temporal selection and visual preparation through availability of first-token logits, including selection, encoding, assembly, and decoder work.Video decoding and processor/tokenizer preprocessing precede the timed interval.Source: §D.2 · p. 15
- Each question receives an untimed same-input warmup followed by a fresh timed forward, with hidden states and caches recomputed.The spatial-selection diagnostic includes the reused prefix and is already part of decoder prefill.Source: §D.2 · p. 15
- 355.18 ms of mean TTFT came from decoder prefill, versus 102.40 ms from vision computation and 14.93 ms from pre-LLM selection/assembly.Mean preprocessing was 750.49 ms; recorded end-to-end latency was 1243.63 ms, excluding the untimed warmup.Source: §D.2 · p. 15
D.3 DEVELOPMENT CONFIGURATIONS
On the Video-MME short development setup, the complete configuration improves accuracy and TTFT over its temporal-only reference. The accompanying token-flow profile shows how many visual states pass through the prefix and later decoder layers.
D.3 DEVELOPMENT CONFIGURATIONS
- 76.89% accuracy at 472.52 ms improves over the temporal-only reference by 0.89 accuracy points and 22.56% lower TTFT.Table 8 covers Qwen3-VL-8B on Video-MME short and includes configurations with different nominal retention ratios, budgets, and execution paths.Source: §D.3 · p. 15, Table 8 · p. 16
D.4 DEVELOPMENT TOKEN FLOW
- The development trace averages 5734.77 visual tokens before compression and 2988.16 after temporal anchoring; 2540.32 states continue through the final 33 decoder blocks.All anchor cells traverse the first three decoder blocks and receive three DeepStack injections; spatial selection retains 85.01% of anchor cells in aggregate, or 44.30% of original visual tokens for the final 33 blocks.Source: §D.4 · p. 16
D.5 COMPRESSION-TIMING AND BUDGET METRICS
The section defines how compression timing and sequence counts are reported, then compares marginal accuracy gains against added TTFT across temporal-retention settings.
D.5 COMPRESSION-TIMING AND BUDGET METRICS
- Reusing the three-block prefix lets its computation guide spatial selection and contribute to the final answer, while later layers process the selected sequence.The timing comparison is between spatial reduction before decoder entry and after the first three decoder blocks; the delayed configuration is the full ContextTom reference.Source: §D.5 · p. 16
- Input tokens count the compressed multimodal sequence, including visual, textual-prompt, and special tokens; retained frames count video frames passed to subsequent visual processing.The input-token total is Nvisual + Ntext + Nspecial.Source: §D.5 · p. 16
- The marginal-gain metric divides the accuracy change between adjacent settings by their TTFT change, using percentage-point accuracy differences.The decoder uses lengths n0 and n1, with full anchor grids in the prefix and spatially selected states afterward.Source: §D.5 · p. 16
- 0.68 points/ms at rt = 0.40→0.52 is about 3.9 times the 0.0258 points/ms at rt = 0.52→0.65.The default setting realizes 73.1% of the sweep's accuracy improvement for 40.9% of its latency increase relative to rt = 0.40; relative to rt = 0.65, it uses 13.7% fewer input tokens and reduces TTFT by 7.3%.Source: §D.5 · p. 16