PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaptToken: Entropy-based Adaptive Token Selection for MLLM Long Video Understanding

Haozhe Qi, Kevin Qu, Mahdi Rad, Rui Wang, Alexander Mathis, Marc Pollefeys

arXiv:2603.28696v1cs.CVcs.AI

TL;DR

Long-video MLLM understanding is constrained by memory and context-length limits, and prior short-clip selection methods lack global relevance comparison and principled stopping. AdaptToken combines cross-modal attention with response entropy for global token allocation, while AdaptToken-Lite uses entropy for early stopping. Across four benchmarks and MLLMs from 7B to 72B, AdaptToken improves accuracy up to 10K frames, and AdaptToken-Lite roughly halves inference time with comparable performance.

  • Problem

    Memory and context-length constraints limit long-video MLLMs, while prior selection methods do not globally compare distant clips or stop after sufficient evidence.

  • Method

    AdaptToken splits videos into frame groups, ranks tokens with cross-modal attention, allocates budgets using response entropy, removes redundant tokens, and supports entropy-based early stopping.

  • Results

    Across four long-video benchmarks and MLLMs from 7B to 72B, AdaptToken consistently improves accuracy and scales to 10K frames, while AdaptToken-Lite roughly halves inference time with comparable performance.

  • Takeaways & Limitations

    Response entropy provides a shared control signal for global token selection and computation reduction in long-video MLLM inference.

  • Takeaways & Limitations

    The approach processes videos in groups because memory and context constraints prevent feeding more than about 1K frames into an MLLM at once.

Abstract

from arXiv · show

Long video understanding remains challenging for Multi-modal Large Language Models (MLLMs) due to high memory costs and context-length limits. Prior approaches mitigate this by scoring and selecting frames/tokens within short clips, but they lack a principled mechanism to (i) compare relevance across distant video clips and (ii) stop processing once sufficient evidence has been gathered. We propose AdaptToken, a training-free framework that turns an MLLM's self-uncertainty into a global control signal for long-video token selection. AdaptToken splits a video into groups, extracts cross-modal attention to rank tokens within each group, and uses the model's response entropy to estimate each group's prompt relevance. This entropy signal enables a global token budget allocation across groups and further supports early stopping (AdaptToken-Lite), skipping the remaining groups when the model becomes sufficiently certain. Across four long-video benchmarks (VideoMME, LongVideoBench, LVBench, and MLVU) and multiple base MLLMs (7B-72B), AdaptToken consistently improves accuracy (e.g., +6.7 on average over Qwen2.5-VL 7B) and continues to benefit from extremely long inputs (up to 10K frames), while AdaptToken-Lite reduces inference time by about half with comparable performance. Project page: https://haozheqi.github.io/adapt-token

1 Introduction

Long-video MLLM understanding is limited by memory and context constraints, while existing frame- and token-selection methods lack global cross-clip relevance assessment. AdaptToken uses response entropy and cross-modal attention to allocate tokens globally, stop early when sufficiently confident, and improve long-video performance.

  • Motivation: Memory demands and context-length limits restrict the resolution and number of frames MLLMs can process in hour-long videos.
  • Motivation: Frame-level selection can retain irrelevant regions, while token-level methods often lack a global criterion for allocating tokens across distant clips.
  • AdaptToken: AdaptToken ranks intra-group tokens with cross-modal attention and uses response entropy to compare group relevance globally.
  • AdaptToken: AdaptToken-Lite uses the same entropy signal to stop processing once the model becomes sufficiently confident, reducing computation.
  • AdaptToken: AdaptToken adds location-aware global token removal to improve token diversity and temporal coverage by suppressing redundancy.
  • Results: Across four long-video benchmarks and multiple MLLMs, AdaptToken consistently improves performance and scales to inputs containing 10K frames.
  • Contributions: AdaptToken is training-free, while its early-stopping variant cuts average inference time by roughly half with comparable or better performance.

2 Related Works

Related work extends MLLMs toward video understanding, but video redundancy and long-context efficiency remain central challenges. Existing approaches select frames or estimate language-model confidence using several signals, motivating AdaptToken’s focus on efficient visual-token selection and model uncertainty.

  • Video MLLMs: Video MLLMs combine visual encoders, projection layers, and LLM backbones to process multimodal token sequences for video reasoning.
  • Long-video selection: Long-video systems extend context windows or apply temporal and spatial pooling, while frame-selection methods use vision-text encoders, text-only LLMs, or learned selectors.
  • Confidence estimation: LLM confidence can be estimated from yes/no-token probabilities, learned confidence tokens, token-level entropy, or self-certainty based on KL divergence.

3 Method

AdaptToken estimates relevance at both group and token levels, using response entropy for global group comparisons and cross-modal attention for intra-group ranking. It allocates tokens across groups, removes redundancy, and supports early stopping when certainty indicates sufficient evidence.

  • AdaptToken overview: AdaptToken splits long videos into frame groups and jointly computes group relevance and intra-group token importance for globally informed selection.It also applies location-aware global token removal and supports early stopping after sufficient evidence is gathered.
  • Group relevance with response entropy: Response entropy is computed from autoregressive next-token distributions, with token certainty defined as the negative entropy and response certainty averaged over the lowest 10% certainty tokens.Higher entropy indicates greater uncertainty; the certainty score quantifies input-frame relevance to the prompt.
  • Group relevance with response entropy: Higher certainty across benchmarks correlates with a higher probability of correct answers, indicating that the corresponding frame group contains prompt-relevant evidence.This provides a training-free quantitative signal for estimating inter-group relevance.
  • Intra-group token importance: Cross-modal attention from a selected late MLLM layer ranks visual tokens within each frame group during the same forward pass used to compute group certainty.The resulting scores capture intra-group token importance while preserving prompt semantics.
  • Entropy-guided global token selection: A global token budget is allocated across groups using certainty scores, so higher-certainty groups retain more top-ranked visual tokens.The allocation uses a softmax temperature fixed to 2, and corresponding positional embeddings are retained.
  • Inference efficiency with early stopping: AdaptToken-Lite stops examining new groups once multiple groups reach sufficiently high certainty, reducing computation while maintaining comparable performance.The method uses group certainty without additional computations to decide when to stop.

4 Experiments

Experiments show that AdaptToken improves long-video understanding across models and benchmarks, scales to very long inputs, and supports substantial efficiency gains through entropy-guided early stopping.

  • Main results: AdaptToken consistently improves the corresponding base MLLMs across four long-video benchmarks and outperforms competing frame/token selection methods.The evaluation covers VideoMME, LongVideoBench, LVBench, and MLVU.
  • Main results: +6.7 average gain over the Qwen2.5-VL 7B baseline is achieved across the four benchmarks, with the largest gains on LVBench and MLVU.AdaptToken also surpasses the previous best result on each benchmark.
  • Main results: +5.7 average gain over the Qwen2.5-VL 72B baseline demonstrates that AdaptToken remains effective for larger MLLMs.The method outperforms all evaluated token-selection methods at this scale.
  • Efficiency with early stopping: AdaptToken-Lite reduces inference time by approximately 50% while maintaining comparable accuracy to AdaptToken on Qwen2.5-VL 7B.The average accuracy difference is −0.7; on VideoMME, time falls from 17.8s to 8.6s per sample.
  • Long-context generalization: +1.8 average improvement on four benchmarks is obtained with Qwen3-VL 8B using up to 4096 frames, while LVBench memory falls from 96 GB to 41 GB and time from 58.5s to 40.7s.This shows benefits even for an MLLM designed for long-context inputs.
  • Scaling to 10K frames: AdaptToken maintains strong performance as input length increases to 10K frames under a fixed memory budget.The method selects prompt-relevant tokens using globally informed allocation and redundancy removal.
  • Certainty measures: Response entropy performs slightly better than alternative certainty measures, although differences are marginal on VideoMME and MLVU.The paper uses existing certainty measures rather than proposing a new certainty metric.
  • Comparison to voting methods: Self-certainty used for globally informed token selection yields larger gains than majority, weighted, or Borda voting used for post-hoc response aggregation.All voting methods improve over the base InternVL2.5 model, but their gains are substantially smaller than AdaptToken’s.

5 Conclusion

AdaptToken is a training-free, model-agnostic framework that uses response entropy to guide global token selection and early stopping. Across models and benchmarks, it improves accuracy on inputs up to 10K frames, while AdaptToken-Lite approximately halves inference time with comparable performance.

  • Method: AdaptToken combines response entropy for global token-budget allocation with cross-modal attention, token diversity, and temporal coverage for informative token selection.AdaptToken-Lite uses the same entropy signal to stop processing once sufficient evidence has been gathered.
  • Results: AdaptToken consistently improves accuracy across four benchmarks and multiple 7B–72B MLLMs while scaling to inputs of up to 10K frames.The conclusion presents this as the paper’s broad empirical result.
  • Efficiency: AdaptToken-Lite cuts average inference time by about half with comparable performance.The efficiency gain comes from entropy-guided early stopping.

A Benchmark details

The evaluation uses four benchmarks spanning diverse tasks and video durations, from short clips to videos averaging more than one hour and often exceeding two hours.

  • VideoMME: VideoMME contains 900 videos and 2,700 multiple-choice questions across 30 subfields, with short, medium, and long duration subsets.Its long subset contains 30–60 minute videos.
  • MLVU: MLVU spans videos from 3 minutes to 2 hours and covers nine tasks including topic reasoning, anomaly recognition, summarization, and plot question answering.The benchmark provides broad variation in video length and task type.
  • LongVideoBench: LongVideoBench contains 3,763 videos and 6,678 questions across 17 categories, emphasizing retrieval and reasoning over specific temporal segments in videos up to one hour.Its questions require analyzing multimodal details tied to particular temporal locations.
  • LVBench: LVBench has an average duration of 4,101 seconds and includes 1,549 questions covering event understanding, key-information retrieval, temporal grounding, and reasoning.It is substantially longer on average than VideoMME and MLVU.

B Implementation details

Evaluations run primarily through LMMS-Eval on eight H100 GPUs, with extended context limits enabling up to 1024 frames for InternVL2.5 and 4096 frames for Qwen2.5-VL.

  • Evaluation setup: Except for the Qwen3-VL GPU-memory test, evaluations use LMMS-Eval on 8 H100 GPUs.The Qwen3-VL memory test is conducted on an H200 GPU.
  • Evaluation setup: Context limits are extended by 16×, reaching 1024 maximum input frames for InternVL2.5 and 4096 for Qwen2.5-VL.The original official settings are 16K tokens for InternVL2.5 and 32K tokens for Qwen2.5-VL.

C Additional experiments

Additional experiments validate AdaptToken’s entropy-based selection and early stopping across backbones, while probing token removal, stopping hyperparameters, and frame-group construction.

  • Cross-backbone early stopping: AdaptToken-Lite matches or slightly exceeds AdaptToken on LongVideoBench and MLVU while requiring only about 65% of inference time with InternVL2.5 8B.The reduction in processed groups closely tracks inference time, identifying group-wise MLLM inference as the dominant runtime cost.
  • Entropy analysis: Correct and incorrect cases remain clearly separable in response-entropy space for Qwen2.5-VL across VideoMME, LongVideoBench, MLVU, and LVBench.The result validates response entropy as a stable uncertainty signal for group selection and supports transferring early-stopping hyperparameters across models.
  • Token-removal ablation: AdaptToken uses a fixed 0.1 token-removal rate, and the ablation evaluates robustness by varying the post-selection removal ratio from 0.05B to 0.3B.The experiment keeps the backbone, token budget B, and inference pipeline fixed while isolating the removal-rate choice.
  • Early-stopping hyperparameters: AdaptToken-Lite’s entropy threshold controls the confidence required for evidence, while its group threshold controls how many confident groups are required before stopping.The ablation varies both hyperparameters and reports MLVU accuracy.
  • Frame-group construction: The frame-group ablation compares video-chunk inputs, which use contiguous local segments, against alternative input organizations on MLVU while keeping the rest of the pipeline unchanged.The study isolates the grouping strategy used for entropy estimation, token allocation, and early stopping.
Loading 2603.28696v1…