PaperlayerView exact revision on arXiv

Source-linked AI summary

ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning

Huanxuan Liao, Zhongtao Jiang, Yupu Hao, Yuqiao Tan, Shizhu He, Ben Wang, Jun Zhao, Kun Xu, Kang Liu

arXiv:2603.28610v2cs.CVcs.AIcs.CL

TL;DR

MLLMs face a costly trade-off between high spatial resolution and long temporal context because visual-token growth follows the encoder’s pixel input. ResAdapt learns per-frame visual budgets before encoding and, across video and image reasoning tasks, improves low-budget efficiency, supporting up to 16× more frames with over 15% performance gains.

  • Problem

    MLLMs struggle to sustain high spatial resolution and long temporal context together because visual-token growth makes video reasoning prohibitively costly.

  • Method

    ResAdapt uses a query-aware per-frame Allocator and CAPO to optimize input-side visual budgets while preserving the MLLM backbone’s native interface.

  • Results

    Across video QA and temporal grounding, ResAdapt advances the efficiency–accuracy frontier, processing up to 16× more frames under equivalent compute with over 15% relative performance gains.

  • Takeaways & Limitations

    ResAdapt shows that selective pre-encoding redistribution of visual budget can improve efficiency while retaining strong performance under aggressive visual-token compression.

  • Takeaways & Limitations

    Transfer beyond video remains uneven, with resizing-based policies failing to deliver uniformly efficiency-preserving gains on image-centric benchmarks.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) achieve stronger visual understanding by scaling input fidelity, yet the resulting visual token growth makes jointly sustaining high spatial resolution and long temporal context prohibitive. We argue that the bottleneck lies not in how post-encoding representations are compressed but in the volume of pixels the encoder receives, and address it with ResAdapt, an Input-side adaptation framework that learns how much visual budget each frame should receive before encoding. ResAdapt couples a lightweight Allocator with an unchanged MLLM backbone, so the backbone retains its native visual-token interface while receiving an operator-transformed input. We formulate allocation as a contextual bandit and train the Allocator with Cost-Aware Policy Optimization (CAPO), which converts sparse rollout feedback into a stable accuracy-cost learning signal. Across budget-controlled video QA, temporal grounding, and image reasoning tasks, ResAdapt improves low-budget operating points and often lies on or near the efficiency-accuracy frontier, with the clearest gains on reasoning-intensive benchmarks under aggressive compression. Notably, ResAdapt supports up to 16x more frames at the same visual budget while delivering over 15% performance gain. Code is available at https://github.com/Xnhyacinth/ResAdapt.

1. Introduction

ResAdapt addresses the prohibitive trade-off between visual fidelity and temporal context by allocating each frame’s visual budget before encoding. Its lightweight Allocator and CAPO training objective preserve the MLLM backbone interface while improving efficiency–accuracy trade-offs.

  • Motivation: Scaling visual fidelity increases visual-token volume, making simultaneous high spatial resolution and long temporal context prohibitive for video reasoning.Reducing resolution can lose answer-determining cues, while shortening clips reduces temporal coverage.
  • Motivation: Downstream token compression and iterative retrieval intervene after encoding, either discarding unrecoverable fine-grained evidence or multiplying backbone inference cost.These approaches accept full-resolution encoder input as a fixed cost, and coarse initial views may undersample cues later targeted by refinement.
  • Method: ResAdapt allocates visual budget per frame before encoding using coarse features and the query, then applies resolution resizing or frame selection while preserving the backbone’s native token interface.The framework targets pixel volume at the encoder input rather than compressing representations after encoding or retrieving them after reasoning.
  • Method: CAPO trains the non-differentiable, continuous allocation policy by converting sparse rollout feedback into a stable asymmetric accuracy–cost signal.The method addresses policy collapse toward minimum budgets and includes a temporal similarity regularizer.
  • Results: 16× more frames are processed at equivalent computational budgets, yielding > 15% relative performance gains while ResAdapt discards over 90% of visual tokens.Evaluations across video QA and temporal grounding show that ResAdapt matches or surpasses state-of-the-art token-economy methods and advances the efficiency–accuracy Pareto frontier.

2. Background and Problem Formulation

The formulation models fixed-fidelity video reasoning as inefficient because visual encoding cost grows quadratically with pixel volume despite sparse temporal evidence. ResAdapt addresses this through a one-step contextual-bandit Allocator that assigns per-frame budgets before backbone decoding and optimizes response quality under visual-cost constraints.

  • 2.1. Preliminaries: Given a text query q and video V = {f_t}^T_{t=1}, the full input is x = (q, V).
  • 2.1. Preliminaries: A fixed-fidelity backbone policy π_ϕ encodes every frame and autoregressively generates rollout y = (y_1, . . . , y_L).The rollout may be represented as a reasoning trace r and final answer o.
  • 2.1. Preliminaries: Visual encoding cost scales quadratically with pixel volume, while evidence needed for complex queries remains sparse in time.
  • 2.1. Preliminaries: The Allocator policy π_θ samples a per-frame allocation vector s with each s_t constrained to [s_min, s_max].It applies a visual budget operator O to each frame, producing transformed frames before backbone encoding.
  • 2.1. Preliminaries: The operator O is agnostic and may implement resizing, frame selection, or other pre-encoding budget controls.
  • 2.2. Problem Formulation: Because allocation occurs once before decoding, the outer problem is a contextual bandit with raw input x as context and continuous vector s as action.Joint training uses the two-stage policy p_θ,ϕ(s, y | x) = π_θ(s | x) π_ϕ(y | x̃).
  • 2.2. Problem Formulation: Response quality is the immediate reward, r(x, s, y) = Q(x, y), while allocation induces visual cost C(s).The ideal objective targets budget τ; Lagrangian relaxation introduces trade-off coefficient λ ≥ 0.
  • 2.2. Problem Formulation: Section 3 instantiates the trade-off with input-side adaptation, CAPO, temporal regularization, and PPO-style surrogate losses, using resize as the concrete operator.Detailed derivations are deferred to Appendix A.1.

3. Method

ResAdapt learns per-frame input resolutions before encoding through a lightweight Allocator while preserving the MLLM backbone’s visual-token interface. Its contextual-bandit training uses CAPO to couple answer correctness with visual cost and regularizers to diversify allocations across redundant frames.

  • Framework: The Allocator predicts a scalar allocation per frame and applies a pre-encoding operator, primarily bilinear resizing, before the video reaches the unchanged backbone.Each frame is transformed as ˜ft = R(ft, st), with st determining its resize factor.
  • Policy formulation: ResAdapt formulates allocation as a contextual bandit whose joint policy factorizes into allocation and response policies, πθ,ϕ(s, y | x) = πθ(s | x) πϕ(y | ˜x).The allocation policy is induced by a latent Beta policy through an affine map, enabling PPO ratios to be evaluated on latent actions.
  • Allocator architecture: The visual encoder is frozen and lightweight, while a shallow Transformer decoder combines temporal self-attention with gated query cross-attention to expose redundancy and query dependence.The resulting per-frame hidden states condition the Allocator’s framewise Beta distributions, whose bounded actions map to scales allowing downscaling and selective upscaling.
  • CAPO: CAPO replaces the raw cost penalty with a dynamic, correctness-dependent signal that encourages low cost on correct rollouts while penalizing compression that harms correctness.Its dynamic pivot interpolates prompt-local and fixed budget references, while asymmetric shaping uses λ− > λ+ > 0 to prevent minimum-budget collapse.
  • Regularization: Temporal-similarity and concentration regularizers differentiate allocations across redundant neighboring frames and preserve stochastic exploration during training.Lsim penalizes joint high-budget allocation for similar adjacent frames, whereas Lcon softly caps Beta concentration.

4. Experiments

Experiments show that ResAdapt’s input-side allocation improves accuracy under severe visual budgets, especially for reasoning-intensive and long-context video tasks, while reducing inference cost. Analyses attribute these gains to sparse, content-driven allocation stabilized by CAPO, with transfer and operator generalization, but also reveal clear limitations.

  • Video QA: ResAdapt achieves 45.7 on VideoMMMU at 11.4% retention, outperforming ToMe (39.2), VisionZip (39.1), FlashVid (39.4), and FixedScale (44.3).The largest gap appears on VideoMMMU, described as the most reasoning-intensive benchmark; the transferred Allocator reaches 56.1 on Qwen3-VL at the same retention.
  • Long-context video: 51.1 on VideoMMMU at 22.9% retention with 128 frames exceeds the 47.9 score of the uncompressed 128-frame model, while 49.2 at 11.1% also surpasses it.These results show that spatial savings can be reinvested into substantially longer temporal context.
  • Temporal grounding: 35.6 Charades-STA mIoU at 16.2% retention exceeds Random Drop, ToMe, FlashVid, and FixedScale scores of 25.7, 26.0, 26.6, and 24.9 at approximately 25–31%.The comparison supports pre-encoding pixel allocation over frame dropping or post-hoc token pruning.
  • Reasoning-time inference: 49.1 Charades-STA mIoU at 16.1% retention restores 128-frame VideoAuto-R1 performance after naive extension drops the score from 41.5 to 28.9.ResAdapt concentrates visual budget on temporally decisive evidence for long-context reasoning.
  • Efficiency: At R≈28%, wall-clock savings reach −59.5% at 128 frames with 78% generation-time reduction.At R≈74%, E2E savings grow to −33.2% at 128 frames, while at R≈51% break-even shifts to 32 frames with −10.6% E2E savings.
  • Mechanism and limitations: The Allocator learns sparse, content-driven temporal allocation, concentrating high resolution around brief informative events rather than applying uniform compression.CAPO stabilizes an intermediate accuracy–cost operating point, Lsim restores frame-level differentiation, and the policy transfers zero-shot to frame selection; however, open-loop allocation can miss brief decisive cues.

5. Related Work

Related work improves multimodal efficiency through input-side adaptation, post-encoding token reduction, output-side agentic reasoning, and reinforcement learning for perception control. ResAdapt differs by learning continuous pre-encoding allocations from task reward and realizing them through operators such as resizing or frame selection.

  • Input-side adaptation before visual encoding: Input-side methods reduce visual cost through temporal downsampling, query-aware frame selection, and iterative search.Early approaches use keyframe selection or clip condensation, while newer methods tailor selection to question types or intermediate evidence.
  • Input-side adaptation before visual encoding: ResAdapt learns continuous input allocations end-to-end from task reward via RL, unlike approaches using predefined resolution tiers and rule-based selection.Its allocations can be realized through different pre-encoding operators, including resizing and frame selection.
  • Model-side token economy after encoding: Model-side methods reduce cost after encoding by pruning, merging, redistributing, or compressing visual tokens in embedding space.Representative techniques include token merging, saliency-guided pruning, progressive dropping, context compression, KV cache sparsity, and diversity-based budgeting.
  • Output-side agentic reasoning: Output-side agentic approaches keep the input fixed and recover efficiency through iterative frame retrieval, regional zooming, and re-querying.These systems range from static predefined cropping or clipping tools to dynamic code-generation primitives exposed through executable interfaces.
  • RL for multimodal reasoning and perception control: RL research for multimodal reasoning and perception control extends language-model post-training to multimodal reasoning and video understanding.Work includes improved advantage estimation, PPO-style stabilization, iterative frame selection, and evidence refinement.

6. Conclusion

ResAdapt addresses video MLLM inefficiency at the input stage by assigning query-aware, per-frame visual budgets before feature extraction. It improves the efficiency-accuracy tradeoff while exposing limitations in open-loop allocation and uneven transfer beyond video.

  • ResAdapt assigns per-frame visual budgets before feature extraction through a query-aware Allocator, restricting encoding to task-relevant sparse inputs.This input-side design leaves native kernel optimizations untouched.
  • ∼10% visual-token retention lets ResAdapt match or exceed dense baselines on complex reasoning tasks.The framework converts spatial savings into expanded temporal coverage under equivalent compute.
  • 16× more frames and >15% relative gains result on long-video benchmarks under equivalent compute.These gains demonstrate the framework’s improved efficiency-accuracy tradeoff at expanded temporal coverage.
  • The current allocator is open-loop and cannot revise resolution budgets during reasoning, while spatial resizing transfers unevenly to non-video modalities.Future work could use intermediate backbone states to trigger selective re-encoding.

Limitations and Future Work · A. Derivations and Theoretical Analysis · A.1. Derivation of Joint RL Formulation

ResAdapt advances the efficiency–accuracy Pareto front but remains limited by front-end overhead, coarse evidence, video-centric validation, and open-loop allocation. Its joint RL derivation models allocation and backbone rollouts as a one-step contextual process and uses alternating, stabilized surrogate optimization.

  • Limitations and Future Work: Fixed pre-encoding overhead offsets downstream attention savings for short sequences, so clear wall-clock speedups primarily emerge in the long-context regime.The overhead includes coarse visual encoding, cross-frame fusion, and distribution prediction; proposed remedies include cached video features, lightweight front-ends, and distilled allocation rules.
  • Limitations and Future Work: Coarse frozen features detect broad redundancy but struggle with small text, subtle objects, and transient answer-critical cues.Multi-scale conditioning, motion-aware features, or lightweight local refinement are proposed to address this gap.
  • Limitations and Future Work: The current resizing-based policy is trained primarily on video tasks, leaving transfer to image-centric benchmarks uneven.Future work includes image–video training mixtures and alternative operators such as hard frame selection.
  • Limitations and Future Work: Open-loop allocation commits budgets before backbone processing and cannot revise low-resolution choices when reasoning or uncertainty signals emerge.Possible extensions include re-encoding, budget revision, or a secondary visual pass triggered by early backbone states.
  • A. Derivations and Theoretical Analysis · A.1. Derivation of Joint RL Formulation: The joint RL formulation treats each video-query pair as a one-step contextual MDP whose episode ends after allocation and response generation.Value functions therefore reduce to immediate rewards, eliminating temporal discounting and complex cross-step credit assignment.
  • A.1. Derivation of Joint RL Formulation: The allocator samples latent actions from a Beta policy, maps them deterministically to continuous allocations, and feeds operator-transformed inputs to the unchanged backbone.The backbone then samples a complete rollout containing a reasoning trace and final answer.
  • A.1. Derivation of Joint RL Formulation: Alternating policy updates avoid compounded joint importance-weight variance by fixing one policy while optimizing the other, with PPO or GRPO using normalized, clipped surrogates.CAPO further replaces the ideal cost-penalized reward with a deliberately biased, cost-shaped group-normalized advantage for lower variance and robust budget control.
  • A.1. Derivation of Joint RL Formulation: Allocator score functions decompose additively across frames, enabling low-variance frame-level credit assignment through practical per-frame clipped objectives.For the backbone, rollout-level advantages are distributed across autoregressive tokens and averaged over sequence length.

A.2. Complexity Analysis · B. Implementation Details · B.1. Training Data

ResAdapt achieves substantial computational savings through adaptive token retention, while preserving standard backbone interfaces and expanding temporal coverage; its training corpus combines filtered Video-Auto-R1 data with 16,500 high-complexity Video-R1 instances.

  • A.2. Complexity Analysis: Adaptive resizing yields a token retention ratio of 0.06–0.16 across the evaluation suite.The learned Beta policy concentrates redundant frames near smin.
  • A.2. Complexity Analysis: The Allocator consumes less than 3% of total inference FLOPs with Pc=14, Lpred=4, and Dpred=1,024.The backbone uses Lmllm=28 layers and Dmllm=3,584 hidden dimensions in this comparison.
  • A.2. Complexity Analysis: Under a strict token budget, adaptive resizing expands the temporal horizon by 1/ρ ≈6–16×.ResAdapt processes T0/ρ frames within the same budget that accommodates T0 full-resolution frames.
  • A.2. Complexity Analysis: Input-side adaptation preserves a standard shorter visual-token sequence, enabling compatibility with FlashAttention, vLLM, and SGLang.Unlike model-side pruning and merging, it avoids irregular token layouts and bespoke kernel fallbacks.
  • B.1. Training Data: The training corpus contains approximately 93.4K samples from difficulty-filtered Video-Auto-R1, retaining image and video data while discarding pure-text examples.The construction also injects 16,500 high-complexity Video-R1 video instances focused on OCR, free-form QA, and regression-style tasks.

B.2. Training Configuration … C. Extended Analysis

The paper specifies training, reward, and prompting details for ResAdapt, then extends analysis across allocation behavior, ablations, qualitative cases, and transfer testing. Reward design combines task-specific metrics, thresholded success indicators, and format validation, while experiments use a standardized reasoning prompt and Qwen2.5-VL-7B setup.

  • B.2. Training Configuration: The Allocator and backbone are trained for one epoch with AdamW, using a global batch size of 128.The cited configuration also specifies distinct learning rates for the two components.
  • B.2. Training Configuration: Learning rates are 2 × 10−5 for the Allocator and 1 × 10−6 for the backbone, with weight decay 0.01 and gradient clipping capped at 1.0.Training constrains the maximum video token budget to 8,192 and samples T=128 frames.
  • B.2. Training Configuration: CAPO samples M=16 allocation trajectories per prompt and executes N=1 rollout per trajectory.Scale factors are bounded within [smin, smax] = [0.2, 1.8], enabling downscaling and selective upscaling.
  • B.3. Reward Design: Reward design defines task-specific metrics for question answering and free-form generation, including binary QA rewards and ROUGE-L for open-ended outputs.Math answers tolerate a 10−2 deviation, multiple-choice answers require exact option letters, and standard QA uses normalized exact matching.
  • B.3. Reward Design: CAPO converts task metrics into a binary success indicator, using direct exact-match outcomes or a strict 0.35 threshold for continuous metrics.The continuous metrics named in the passage are ROUGE-L, temporal IoU, and Grounding QA.
  • B.4. Prompt Template: Format validation requires exactly one reasoning block and one answer block, with the final answer inside boxed notation; malformed outputs receive a 0.2-weight penalty.The required tags are <think>...</think> and <answer>...</answer>.
  • C. Extended Analysis: Extended analysis examines allocation policies by benchmark, duration, and category, plus temporal-regularizer and reward-design ablations, qualitative cases, and beyond-video transfer.Unless otherwise noted, plots use Qwen2.5-VL-7B processing 32 uniformly sampled frames.

C.1. Extended Scale Policy Analysis

ResAdapt adapts visual allocation to benchmark, clip length, and spatial complexity rather than applying fixed compression. Its success is associated with concentrating resolution on decisive frames, while preserving 89% of originally correct predictions under compression but retaining comparable error correction and induction rates.

  • Benchmark-level budget allocation: Reasoning-intensive tasks receive higher mean scales than perception-oriented tasks, 0.435 vs. 0.417, with MMMU-Adaptation highest and VideoMME lowest.The policy operates without benchmark identity during training and calibrates to anticipated visual complexity rather than applying fixed compression.
  • Long-context behavior: As clip duration grows, mean scale decreases from 0.342→0.336→0.332 while within-video diversity increases from 0.085→∼0.095.Longer videos are compressed more aggressively overall but allocated with greater selectivity.
  • Category-level allocation: Within VideoMME, Sports Competition receives maximum budget and Artistic Performance minimum, tracking spatial complexity rather than task difficulty.Sports Competition is described as dense and high-motion, while Artistic Performance is visually sparse.
  • Selectivity and success: Correct predictions consistently exhibit higher frame-level selectivity, peaking on MMMU-P, indicating sharper concentration on decisive frames rather than larger average budgets.Selectivity is measured with the Gini coefficient of predicted scales, where higher values indicate concentrated budget on a sparse subset of frames.
  • Robustness and failure modes: 89% of originally correct samples survive compression, while error correction and error induction rates remain comparable.Selective redistribution can rescue failures by magnifying critical details but may also destroy fine-grained evidence when the decisive cue is transient or visually incomplete.

C.2. Extended Ablation Studies · C.3. Qualitative Case Studies

Extended ablations show that Lsim and CAPO are necessary for selective, content-adaptive allocation: Lsim restores temporal scale diversity, while CAPO avoids degenerate high- or low-scale solutions. Qualitative cases show task-dependent evidence localization, but also reveal brittleness when the policy misses a brief decisive cue.

  • C.2. Extended Ablation Studies: 4×–693× within-video variation returns when Lsim is activated, whereas removing it collapses scale diversity to near-zero across benchmarks.Without Lsim, σ < 0.003; CAPO controls the global budget level while Lsim breaks the uniform-scale equilibrium.
  • C.2. Extended Ablation Studies: With the regularizer active, scale distributions become bimodal, per-video ranges expand, adjacent-frame variation increases, and the Gini coefficient rises.These changes mark a transition from degenerate uniform allocation to genuinely selective allocation.
  • C.2. Extended Ablation Studies: Direct cost penalties collapse per-sample scale range to zero, while cost-free training reaches a uniform high-scale plateau; CAPO preserves validation adaptivity.Figure 15 tracks smax −smin across training.
  • C.2. Extended Ablation Studies: Accuracy-only training saturates near smax, direct cost optimization collapses to smin, and CAPO converges to a stable intermediate operating point.The baselines are stable but stabilize at degenerate extremes, whereas CAPO preserves content-adaptive allocation.
  • C.3. Qualitative Case Studies: Qualitative analyses render 32 uniformly sampled frames at predicted scales, with warmer borders indicating aggressive upscaling.Four analyses map allocation behavior to reasoning outcomes across Figures 17–20.
  • C.3. Qualitative Case Studies: In Video-MMMU, comprehension triggers sparse allocation around diagram-heavy slides, while adaptation allocates differently for a dense numeric table requiring χ2 computation.Both tasks come from identical educational domains but induce markedly different allocation strategies.
  • C.3. Qualitative Case Studies: VideoMME analysis shows precise localization of brief date overlays and aggressive compression of repetitive sky footage.The accompanying duration analysis reports lower average scale, greater within-video diversity, and lower accuracy for longer clips.
  • C.3. Qualitative Case Studies: A brief fork cue exposes a failure mode: the policy upscales an adjacent frame and compresses the critical frame, destroying fine-grained evidence.Allocation also varies across VideoMME categories, assigning larger budgets to crowded motion or finer local evidence rather than simply easier categories.

C.4. Boundary-Case Transfer Beyond Video

ResAdapt’s input-side allocation transfers across video tasks and operators but does not reliably preserve efficiency on dense static-image benchmarks when trained only on video. Reliable static-image handling requires explicit joint training, while diagnostics show that effective adaptation depends on selective, non-uniform resolution allocation.

  • Boundary-Case Transfer Beyond Video: Video-trained ResAdapt occasionally identifies images needing high fidelity, such as ChartQA, but does not consistently improve efficiency on dense static-image benchmarks.The image-transfer evaluation is exploratory and reports per-task retention ratios in Table 7.
  • Boundary-Case Transfer Beyond Video: Input-side allocation generalizes across video tasks and operators, whereas a strictly video-trained policy requires explicit joint training for reliable static-image distributions.The limitation concerns the training distribution rather than the allocation mechanism itself.
  • Boundary-Case Transfer Beyond Video: Adding Lsim restores broad within-video scale variation across all five benchmarks, avoiding the near-zero diversity produced without the regularizer.The regularizer increases per-video scale standard deviation σ and supports selective allocation.
  • Boundary-Case Transfer Beyond Video: ResAdapt concentrates resolution on answer-bearing frames, such as diagrams, numeric tables, and decisive overlays, while compressing irrelevant or homogeneous context.The case studies also show a failure mode: enlarging a nearby frame can lose the decisive fine detail when the actual evidence frame is compressed.
Loading 2603.28610v2…