Source-linked AI summary
TeQCo: Temporal Quota Consistency for Training-Free Video Token Compression
Yuqi Li, Xianglong Wang, Neil Rodrigues, Yiru Fang, Weilun Feng, Chuanyue Yan, Chuangguang Yang, Yingli Tian, Zhongwei Cheng, Wei Li, Lin Chen
TL;DR
Video-token compression must preserve temporal context and question-relevant evidence, but sequential stages can conflict when global ranking removes positions assigned capacity upstream. TeQCo resolves this by carrying content-adaptive frame quotas into question-dependent selection. Across four backbones, its compact sequences enable faster first-token delivery while retaining useful task evidence.
Problem
Video-token compression must control computation while retaining answer-relevant evidence, but a global ranking stage can overwrite upstream temporal allocation and remove a position entirely.
Source: §1 · p. 1Method
TeQCo builds content-adaptive frame candidate sets, then uses prior-corrected task attention to select candidates within inherited integer frame quotas.
Source: §1 · p. 2Results
Across four backbones, TeQCo’s compact sequences enable faster first-token delivery while retaining useful task evidence.
Source: §5 · p. 10Takeaways & Limitations
The results illustrate that successive stages can specialize in different signals while retaining a shared temporal allocation.
Source: §5 · p. 10Peak memory decreases for the 8B model but increases slightly for the 2B model, making latency reduction the clearest benefit of the current execution path.
Source: §4.3 · p. 8
Abstract
from arXiv · showhide
Video token compression must control both temporal coverage and task relevance, but sequential stages can conflict: a content-driven allocation stage may be overwritten by a global ranking that removes entire temporal positions. We identify this as an allocation conflict and propose TeQCo, a training-free framework that resolves it through temporal quota consistency. TeQCo allocates candidate capacity from first-order changes in normalized frame representations and aggregates all tokens within each frame into a compact candidate set. Candidates are then scored by prior-corrected task attention and selected within integer frame quotas inherited from the first-stage counts. Content variation determines temporal capacity; question relevance determines which candidates fill it. This design guarantees an exact token budget and a per-frame coverage floor whenever feasible. On the 2,700-question Video-MME benchmark with Qwen3-VL-8B, TeQCo achieves 68.52% accuracy at 510 ms TTFT, a 32.9% TTFT reduction with a 1.00-percentage-point accuracy decrease.
1 INTRODUCTION
Video-token compression must preserve both changing temporal context and question-relevant evidence, yet sequential stages can conflict when global ranking removes positions assigned capacity upstream. TeQCo coordinates the stages by making content variation determine frame capacity and question relevance determine which candidates fill it.
1 INTRODUCTION
- A content-driven stage retains (6, 4, 2) tokens across three temporal positions; subsequent unconstrained global Top-K selection reducing the total from 12 to 6 can yield (5, 1, 0), removing the third position entirely.Source: §1 · p. 1
- For relational questions, a quieter reference moment can supply temporal meaning alongside a salient frame containing strong local evidence.The motivation is that determining whether a state change occurs before or after an action may require both moments.Source: §1 · p. 2
- TeQCo allocates candidate capacity from first-order changes in normalized frame representations, then uses prior-corrected task attention to select within inherited quotas.Content variation sets temporal capacity; question relevance chooses the evidence retained within each frame.Source: §1 · p. 2
- Temporal quota consistency converts upstream candidate proportions into bounded integer quotas, enforcing the exact final budget and a per-frame coverage floor whenever feasible.The example assigns (3, 2, 1), retaining all three positions while halving the candidate sequence.Source: §1 · p. 2
- TeQCo inherits integer frame quotas from realized candidate counts, making temporal capacity a shared constraint across sequential compression stages.This addresses the conflict in which global Top-K can remove a temporal position that an upstream content-driven stage retained.Source: §1 · pp. 1–2
2 RELATED WORK
Prior video-token compression methods use complementary signals for token similarity, temporal redundancy, frame budgeting, and task relevance. TeQCo connects these signals through inherited frame quotas that constrain downstream selection.
Visual token reduction
- Visual-token methods use feature similarity, visual attention, or language-model attention to merge or prune tokens, while TeQCo connects content and task signals through shared temporal allocation.The surveyed approaches include ToMe, LLaVA-PruMerge, VisionZip, FastV, and SparseVLM.Source: §2 · p. 2
Temporal redundancy and adaptive budgets
- Temporal compression methods exploit adjacent-frame similarity, temporal segmentation, frame uniqueness, or semantic-trajectory curvature to organize unequal frame budgets.TeQCo carries realized candidate proportions into question-dependent selection and prevents a later global top-K from reducing an allocated frame to zero.Source: §2 · pp. 2–3
Task relevance and multi-stage coordination
- Task-relevance methods combine attention or staged merging and pruning, while TeQCo uses realized candidate counts as an explicit interface between content allocation and question-dependent selection.Frame-wise ranking chooses the highest-scoring evidence within each assigned quota, so the two signals govern distinct decisions.Source: §2 · p. 3
3 METHOD
TeQCo compresses frozen vision-language-model visual tokens in two stages: content-adaptive candidate construction followed by question-dependent selection within inherited frame quotas. The shallow scoring pass is followed by a fresh full-depth prefill over the selected visual tokens and preserved nonvisual text.
3.1 OVERVIEW
- TeQCo takes the visual tokens from a frozen vision-language model and targets K ≪ N visual tokens for language-model prefill while keeping all model parameters frozen.The input has T processed temporal positions and P spatial tokens per position, for N = TP visual tokens; nonvisual tokens are denoted M.Source: §3.1 · p. 3
3.2 CONTENT-ADAPTIVE CANDIDATE CONSTRUCTION
- Frame capacity is allocated from first-order changes between normalized mean frame features, with standardized displacements converted into integer candidate counts under a nominal budget.The realized counts account for bounds on each frame’s candidate count and define the capacity inherited by Stage 2.Source: §3.2 · p. 3
- Within each frame, local temporal differences choose representative locations, and remaining tokens join their most similar representative so every original token contributes to a candidate cluster.Cluster aggregation incorporates remaining visual context alongside the changing regions prioritized by temporal differences.Source: §3.2 · pp. 4–5
3.3 QUOTA-CONSTRAINED TASK SELECTION
- A shallow pass contrasts task-conditioned attention with a question-free reference to score candidates, then selects the top q_t candidates within each frame’s inherited quota.The quotas carry realized candidate proportions forward; a feasible coverage floor retains at least one token at each nonempty temporal position.Source: §3.3 · p. 5
- For fixed scores and quotas, frame-wise top-K selection exactly maximizes the additive relevance objective while satisfying the final budget and frame capacities.The selected indices also govern embeddings, auxiliary visual features, and positions; all nonvisual text is preserved and the probe is removed.Source: §3.3 · p. 5
3.4 EXECUTION AND COMPUTATIONAL BENEFIT
- TeQCo performs shallow scoring at the first three decoder blocks, then runs a fresh full-depth prefill on K + M tokens rather than N + M.The default α = β = 0.5 targets about 25% visual retention; τ = 8 controls content adaptation and m = 1 sets the coverage floor.Source: §3.4 · pp. 5–6
4 EXPERIMENTS
Across four backbones and four benchmarks, the experiments assess accuracy and latency, total execution costs, component composition, and duration sensitivity. TeQCo lowers TTFT with limited Video-MME accuracy changes across backbones, while total-latency and duration effects vary by workload.
4.1 EXPERIMENTAL SETUP
- The evaluation covers Video-MME, EgoSchema, LongVideoBench, and MVBench with four backbones, using backbone-specific sampling, compression, and latency-measurement settings.Video-MME has 2,700 questions; TTFT includes visual encoding, compression, shallow scoring, selection, and final prefill, while E2E also includes preprocessing and answer generation.Source: §4.1 · p. 6
4.2 RQ1: TASK PERFORMANCE AND LATENCY
- Tables 1–2 show lower TTFT with limited changes in full Video-MME accuracy across all four backbones; both Qwen models jointly improve accuracy and latency over several compression baselines.InternVL shows a strong tradeoff, while LLaVA has closer competition among operating points.Source: §4.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7
4.3 RQ2: INFERENCE LATENCY AND MEMORY
- TTFT improves in every listed full-input comparison, but Video-MME has a small E2E increase despite faster TTFT because runtime outside the first-token interval outweighs the saving.Peak memory decreases for the 8B model and increases slightly for the 2B model; the authors identify latency reduction as the clearest benefit of the current execution path.Source: §4.3 · p. 8, Table 3 · p. 8
4.4 RQ3: CONTRIBUTIONS OF CANDIDATE CONSTRUCTION AND FINAL SELECTION
- Candidate variants all lower TTFT relative to full input, but their accuracy ordering varies by duration; full TeQCo is faster than first-order candidates at every duration, with improved short- and medium-video accuracy and nearly unchanged long-video accuracy.First-order allocation complements curvature by responding to sustained content variation; Table 4 comparisons change the token budget when quota-constrained selection is added.Source: §4.4 · pp. 8–9, Table 4 · p. 9
4.5 RQ4: SENSITIVITY TO VIDEO DURATION
- Both Qwen models retain TTFT gains across short, medium, and long videos, but relative savings decrease with duration; accuracy changes are also non-monotonic, particularly for Qwen3-VL-2B.With fixed 32-frame sampling, InternVL and LLaVA have closely grouped TTFT values across duration subsets.Source: §4.5 · p. 9
5 CONCLUSION
TeQCo shares temporal capacity across sequential compression stages, using content-adaptive aggregation to form candidates and inherited integer quotas to guide question-dependent selection. The paper reports faster first-token delivery while retaining useful task evidence across four backbones.
5 CONCLUSION
- TeQCo uses inherited integer quotas to guide question-dependent selection from a compact, content-adaptive candidate pool.The framework makes temporal capacity a shared constraint across sequential compression stages.Source: §5 · p. 9
- The framework combines exact budget control, feasible temporal coverage, and optimal frame-wise selection for the fixed-score objective.Source: §5 · p. 9
- Experiments across four backbones show faster first-token delivery while retaining useful task evidence.Source: §5 · p. 10
AI USE STATEMENT
ChatGPT assisted with manuscript organization, language polishing, and LaTeX formatting, as well as interpreting author-provided experimental results and formulating a score-calibration property and proof. The authors take responsibility for the final manuscript and its claims, arguments, and results.
AI USE STATEMENT
- ChatGPT assisted with manuscript organization, language polishing, and LaTeX formatting.Source: AI USE STATEMENT · p. 10
- ChatGPT also assisted in interpreting author-provided results and formulating the frame-wise score-calibration property and its proof.Source: AI USE STATEMENT · p. 10
- The authors take full responsibility for the final manuscript, including its scientific claims, mathematical arguments, and experimental results.Source: AI USE STATEMENT · p. 10
A.1 NORMALIZATION, BOUNDARY CASES, AND CANDIDATE COUNTS
The appendix specifies normalization and boundary handling for candidate counts, then describes how within-frame clusters are formed and aligned across feature streams and positions.
A.1 NORMALIZATION, BOUNDARY CASES, AND CANDIDATE COUNTS
- For T > 1, the first displacement is initialized from the mean of subsequent displacements; for T = 1, the single-frame budget is allocated directly.The appendix sets ϵ = 10^-6 and uses sample standard deviation in Equation (2).Source: §A.1 · p. 12
- Candidate counts are rounded by largest fractional remainder, then clipped to between one and P representatives per frame.Realized counts are retained after clipping, so active bounds can make their total differ from nominal B1.Source: §A.1 · p. 12
A.2 CLUSTER ASSIGNMENT AND FEATURE ALIGNMENT
- Each representative forms its own cluster, and Equation (4) averages the original features assigned to that cluster.Temporal comparisons choose representative locations, while aggregation combines features within a frame.Source: §A.2 · p. 12
- The shared assignment map aligns main visual embeddings with all DeepStack feature streams.Source: §A.2 · p. 12
- Candidates remain in frame order, use representative grid indices for multimodal positions, and share retained indices across embeddings, auxiliary features, and position encodings.The second stage directly prunes candidates; aggregation occurs once during candidate construction.Source: §A.2 · p. 12
A.3 PROBE ATTENTION AND TASK SCORES
The appendix describes probe-based task scoring and exact integer frame quotas, then establishes optimal frame-wise selection and invariance to frame-wise score calibration. It also specifies the scoring and inference sequence.
A.3 PROBE ATTENTION AND TASK SCORES
- A newline-token probe is placed after the final visual token, and causal attention from the probe and subsequent text-query positions supplies candidate scores.The text suffix includes the question, answer options, and template markers; the probe can access the preceding video and prompt but not the later question.Source: §A.3 · p. 12
A.4 EXACT INTEGER QUOTA ALLOCATION
- The quota procedure adjusts integer frame quotas within lower-bound and candidate-count limits until their total equals K.With m = 1 and sufficient budget, every nonempty processed temporal position retains a candidate.Source: §A.4 · pp. 12–13
A.5 OPTIMAL SELECTION FOR FIXED SCORES AND QUOTAS
- For fixed scores and quotas, selecting the qt highest-scoring candidates in each frame is optimal for the stated objective.Exchanging a selected lower-scoring candidate for an unselected higher-scoring candidate in the same frame improves the objective without changing the quota.Source: §A.5 · p. 13
A.6 INVARIANCE TO FRAME-WISE SCORE CALIBRATION
- With candidates and quotas fixed, positive scaling and an additive offset applied within a frame preserve its candidate ordering and selected set.The property applies to the final selection operator with fixed candidates and quotas.Source: §A.6 · p. 13
B.1 COMPRESSION AND INFERENCE PROCEDURE
- The scoring forward evaluates decoder blocks one through three and reads attention at the third block’s input; final prefill restarts at the first block after selection.Visual encoding is completed before candidate construction.Source: §B.1 · p. 13
B.2 COST DECOMPOSITION AND TIMING BOUNDARY
TeQCo’s cost accounting includes candidate construction, scoring, quota selection, and final prefill, while its final prefill processes a shorter sequence than the baseline. TTFT covers the combined compression and inference path, excluding video decoding and processor preprocessing.
B.2 COST DECOMPOSITION AND TIMING BOUNDARY
- The cost decomposition includes language-model processing by sequence length, quota selection, and final processing of K visual plus M nonvisual tokens.The approximate-cost expression also includes a quota/select term.Source: §B.2 · p. 13
- TeQCo shortens the full-decoder sequence from N + M to K + M, with net latency savings counted after candidate construction, shallow scoring, and quota-based selection.Within-frame matching has principal similarity cost proportional to P t(P −nt)ntd.Source: §B.2 · p. 13
- Inference accounting includes indexing, per-frame CPU/GPU transfers, and shallow decoder computation; sequence-length memory cost is quadratic.The default configuration uses first-order content-change ordering, α = 0.5 candidate retention, β = 0.5 final retention, and a feasible minimum of one frame token.Source: §B.2 · p. 14
- The reported TTFT includes visual encoding, candidate construction, shallow scoring, quota selection, and final prefill, but excludes video decoding and processor preprocessing.It therefore measures the combined compression and inference path.Source: §B.2 · p. 14
C.1 DEFAULT CONFIGURATION
The default retention settings target about one-quarter of visual tokens for final prefill. The allocation diagnostic measures how final temporal counts depart from realized candidate proportions, including effects of integer counts and coverage bounds.
C.1 DEFAULT CONFIGURATION
- αβ = 25% targets an approximately 75% reduction in the visual portion of the final-prefill input, while nonvisual text remains in the sequence.Realized counts can differ because of first-stage clipping and integer allocation; scoring computation is included in the cost accounting.Source: §C.1 · p. 14
C.2 DEVIATION FROM THE INHERITED ALLOCATION
- The temporal-allocation diagnostic is total-variation distance between the final temporal distribution and realized candidate proportions.A value of zero indicates exact proportional inheritance.Source: §C.2 · p. 15
- Positive diagnostic values quantify adjustment associated with integer counts and active coverage bounds; final frame counts and empty-frame fraction make allocation changes measurable.Source: §C.2 · p. 15