PaperlayerView exact revision on arXiv

Source-linked AI summary

SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models

Ahmadreza Jeddi, Enming Zhang, Jasper Gerigk, Hakki Karaimer, Mozhgan Nasr Azadani, Jiayun Luo, Minh Ngoc Le, Gholamali Aminian, Hugo Buurmeijer, Yongchao Chen, Leonid Sigal, Igor Gilitschenski, Konstantinos G. Derpanis, Marco Pavone, Babak Taati

arXiv:2609.34044v1cs.CV

TL;DR

Aggressive visual-token pruning reduces inference cost but can sharply degrade performance, commonly attributed to lost visual evidence. SCOPD adapts reasoning from sparse contexts using full-context teacher supervision, and at 10% retention improves benchmark performance over the Vanilla model; SCOPD+ improves it further.

  • Problem

    Aggressive visual-token compression can sharply degrade VLM performance, commonly attributed to removing task-relevant visual evidence.

    Source: §1 · p. 1, §3.1 · p. 4

  • Method

    SCOPD distills from a full-context teacher on student-generated sparse-context prefixes, while SCOPD+ selectively supervises visually sensitive positions identified by a small visual-budget intervention.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    At 10% visual-token retention across 13 image benchmarks, Vanilla retains 86.37% of unpruned performance, compared with 90.49% for SCOPD and 92.43% for SCOPD+.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across token budgets, pruning operators, model families, and image and video benchmarks, both methods improve sparse-context reasoning without architectural changes or additional inference-time computation.

    Source: §5 · p. 9

    Experiments focus on controlled sparse-context adaptation with VisionZip as the primary training-time pruning operator and fixed retention budgets, and primarily evaluate short-form image and video reasoning.

    Source: §5 · p. 10

Abstract

from arXiv · show

Reasoning vision-language models (VLMs) process images and videos as long sequences of visual tokens, making inference expensive. Training-free token pruning reduces this cost, but aggressive compression can sharply degrade performance, often attributed to irreversible loss of task-relevant visual information. We show that this explanation is incomplete. In a fixed-context Pass@K analysis, repeated sampling from the same pruned visual representation recovers many examples missed by greedy decoding, indicating that useful visual evidence can remain accessible but be used unreliably. We call this the representation-utilization gap. Motivated by this observation, we introduce SCOPD, a sparse-context on-policy self-distillation framework in which a student generates reasoning trajectories from pruned visual tokens while a privileged full-context teacher supervises the same on-policy prefixes. SCOPD requires no ground-truth responses, architectural changes, or additional inference-time computation. We further introduce SCOPD+, which uses a small visual-budget intervention to identify visually sensitive response positions and selectively distill them. At 10% visual-token retention, the Vanilla model retains 86.37% of its unpruned performance across 13 benchmarks. SCOPD raises this to 90.49%, while SCOPD+ further improves it to 92.43%. Across token budgets, benchmarks, and pruning operators, our results show that efficient reasoning depends not only on which visual information survives pruning, but also on how reliably the model learns to use it.

1 INTRODUCTION

Aggressive visual-token pruning can leave useful evidence accessible but unreliable to use; SCOPD and SCOPD+ adapt reasoning to sparse contexts, improving performance without architectural changes or additional inference-time computation.

1 INTRODUCTION

  • VLM inference costs rise with hundreds or thousands of visual tokens, while aggressive compression can sharply reduce performance and is commonly attributed to lost task-relevant information.Source: §1 · p. 1
  • Repeated sampling recovers correct trajectories from the same pruned visual context, suggesting that evidence can remain accessible but be used unreliably during reasoning.The paper calls this mismatch the representation–utilization gap.Source: §1 · p. 2
  • SCOPD trains on student-generated reasoning under sparse visual context while a full-context teacher supervises the same on-policy prefixes.It needs no ground-truth responses or reasoning traces, architectural changes, or additional inference-time overhead.Source: §1 · pp. 2–3
  • SCOPD+ uses a small visual-budget intervention to identify visually sensitive response positions and selectively distills them.It backpropagates through only a fraction of response positions.Source: §1 · pp. 2–3
  • At 10% visual-token retention across 13 image benchmarks, Vanilla retains 86.37% of unpruned performance, versus 90.49% with SCOPD and 92.43% with SCOPD+.The methods also generalize across pruning operators, model families, and from image training to video evaluation without video-specific post-training.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work reduces visual context through pruning, merging, and compression, and studies distillation for adapting models to compressed inputs. SCOPD instead tests whether evidence in a deliberately sparsified conditioning representation remains usable.

Efficient Visual Contexts for VLMs

  • Visual-token reduction methods prune or merge tokens using attention, similarity, diversity, graph structure, or language-conditioned relevance, while learned sparsification adds training or architectural changes.The literature also includes progressive token- and layer-level consistency distillation for compressed representations (Wen et al. 2026b).Source: §2 · p. 3

On-Policy and Privileged Self-Distillation

  • On-policy distillation addresses student-generated state distributions with teacher supervision on student trajectories; multimodal variants use privileged visual views or evidence.SCOPD keeps the model unchanged and uses its corresponding unpruned representation as privileged supervision for a sparsified conditioning representation.Source: §2 · p. 3

Selective and Efficient On-Policy Distillation

  • Selective distillation methods use uncertainty, teacher–student disagreement, or trajectory selection, whereas SCOPD+ perturbs visual evidence while holding the reasoning prefix fixed.That intervention distinguishes positions sensitive to missing visual context from ordinary language-level variation.Source: §2 · pp. 3–4

Recoverability Beyond Greedy Decoding

  • Pass@K work shows greedy decoding can underestimate recoverable capabilities, and prior work finds structurally pruned language models may retain substantial Pass@K (Zhang et al. 2026b; Wen et al. 2026a).SCOPD instead prunes the conditioning signal and holds the model and sparse visual representation fixed across generations.Source: §2 · p. 4

3 ADAPTING VLMS TO SPARSE VISUAL CONTEXTS

Fixed-context sampling shows that correct reasoning can remain recoverable from pruned visual tokens even when greedy decoding fails. SCOPD adapts the model to sparse context with full-context teacher supervision, while SCOPD+ selectively targets positions responsive to visual evidence.

3 ADAPTING VLMS TO SPARSE VISUAL CONTEXTS

  • The section frames pruning failures as potentially reflecting unreliable use of surviving evidence, not only its removal.The motivating question is whether a correct trajectory can be recovered from the exact same sparse visual context.Source: §3.1 · p. 4, §3 · p. 4

3.1 MOTIVATION: THE REPRESENTATION–UTILIZATION GAP

  • At 10% retention, greedy success is 53.2% while Pass@64 is 79.6% from the same sparse representation.Language-only success rises only from 2.8% to 4.2%.Source: §3.1 · p. 4

3.2 SCOPD: SPARSE-CONTEXT ON-POLICY SELF-DISTILLATION

  • SCOPD trains on student-generated trajectories from sparse context, with a full-context teacher scoring the same prefixes for token-level supervision.This aligns training with states visited by the pruned model without requiring ground-truth answers or reasoning traces.Source: §3.2 · p. 5

3.3 SCOPD+: SELECTIVE DISTILLATION OF VISUALLY SENSITIVE TOKENS

  • SCOPD+ uses JSD between student distributions under nearby visual budgets to select visually sensitive positions for full-context supervision.It preserves SCOPD’s on-policy trajectories while restricting backpropagation to selected positions.Source: §3.3 · p. 6

4 EXPERIMENTS

Across 13 image benchmarks, SCOPD and SCOPD+ improve sparse-context performance, with SCOPD+ strongest under aggressive pruning. Ablations test selective supervision, transfer across pruning operators, models and video, hyperparameters, and training overhead.

4.1 EXPERIMENTAL SETUP

  • The experiments use Qwen2.5-VL-7B-Instruct with VisionZip, a 10% default visual-token budget, and LLM-only LoRA; SCOPD methods learn from a full-context teacher.Evaluation covers 13 image benchmarks and multiple visual-token budgets; SFT and supervised baselines use reference traces and answers.Source: §4.1 · pp. 6–7, §4.2 · p. 7

4.2 MAIN RESULTS

  • At 10% retention, Avg13 rises from Vanilla’s 86.37% to SCOPD’s 90.49% and SCOPD+’s 92.43%; at 20%, SCOPD+ reaches 97.25%.At 20% retention, SCOPD improves Avg13 over Vanilla from 93.42% to 96.80%.Source: Table 1 · p. 7, §4.2 · p. 7
  • SCOPD and SCOPD+ remain essentially unchanged with full visual context, at 100.67% and 100.02% Avg13, while their advantage grows as the visual budget shrinks.The authors interpret this pattern as evidence that post-training helps models use compressed visual evidence more reliably.Source: §4.2 · p. 7

4.3 ABLATIONS

  • At 10% retention, SCOPD and SCOPD+ raise Qwen3-VL Avg6 from 75.29% to 81.6% and 82.92%, and video Avg5 from 90.88% to 95.70% and 96.60%.The video evaluation uses models post-trained on images, without video-specific training.Source: Table 4 · p. 8, §4.3 · p. 8, Table 5 · p. 8
  • Without further adaptation, VisionZip-trained SCOPD+ improves Avg6 under DivPrune (83.47% to 91.23%), random pruning (82.06% to 88.85%), and FastV (81.47% to 86.03%).The authors highlight transfer to FastV, where pruning occurs inside the language model.Source: Table 3 · p. 8, §4.3 · p. 8
  • With ρ = 10% of response positions, SCOPD+ leads Avg6 at 95.25%, while selecting the least visually sensitive positions lowers it to 89.13%.It outperforms dense SCOPD (94.44%), Top-KL (94.51%), TIP (94.21%), and random selection (93.56%).Source: §4.3 · p. 8, Table 2 · p. 8
  • SCOPD+ performance peaks at ρ = 10% and δ = 1%; at the 10% deployment budget, the small intervention probes local sensitivity to marginal visual evidence.Denser supervision provides no additional benefit, and larger interventions compare against substantially richer visual representations.Source: §4.3 · pp. 8–9, Figure 3 · p. 9
  • Forward and reverse KL perform similarly (92.43 vs. 92.39 Avg13), but using current student weights as teacher collapses to 12.35; SCOPD+ adds 1.9% measured training time.The default uses forward KL with an EMA teacher; theoretical compute increases by 21.4% over SCOPD.Source: §4.3 · p. 9, Table 6 · p. 9

5 CONCLUSION

The conclusion identifies a representation–utilization gap and reports that SCOPD and SCOPD+ improve sparse-context reasoning across settings without architectural changes or additional inference-time computation.

5 CONCLUSION

  • Both methods improved sparse-context reasoning across token budgets, pruning operators, model families, and image and video benchmarks without architectural changes or extra inference-time computation.SCOPD adapts to sparse visual contexts using a privileged full-context teacher; SCOPD+ selectively focuses on visually sensitive positions.Source: §5 · p. 9
  • Experiments focused on VisionZip as the primary training-time pruning operator, fixed retention budgets, and primarily short-form image and video reasoning.The authors suggest studying other pruning strategies and longer-horizon video, embodied, and agentic tasks.Source: §5 · p. 10

REPRODUCIBILITY STATEMENT

  • Training and evaluation configurations, data processing, prompts, pruning settings, baselines, compute measurements, ablations, and training code are documented for reproducibility.The fixed-context Pass@K protocol, validity-judge prompt, and GRPO setup are also documented.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

  • Generative AI tools assisted with methodology and experimental design, results interpretation, manuscript drafting and editing, and code and figure/table work; authors reviewed outputs and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A EXPANDED FIXED-CONTEXT PASS@K ANALYSIS

The fixed-context analysis tests whether repeated reasoning can recover correct solutions from unchanged pruned visual tokens. It evaluates recoverability while checking that answers are supported by valid visual reasoning.

Diagnostic set and generation setup

  • The diagnostic set contains 500 free-form numerical image–question pairs from six sources, with 493 distinct images, selected from examples solved by unpruned greedy generations.The pruned representation is computed once and kept fixed across generations, which differ only in the language-generation trajectory.Source: A EXPANDED FIXED-CONTEXT PASS@K ANALYSIS · p. 17

Reasoning-aware validity evaluation

  • A semantic judge checks answer correctness and whether decisive observations and operations are supported by the image and question, rejecting unsupported guesses and answer-critical hallucinations.Uncertain cases are counted as failures; Figure 5 illustrates examples where answer correctness alone can miss invalid reasoning.Source: Figure 5 · p. 18, A EXPANDED FIXED-CONTEXT PASS@K ANALYSIS · p. 19

Pass@K metric

  • Pass@K uses the number of valid responses among 64 stochastic generations, while Hit≥4@64 measures the fraction of examples with at least four valid generations.Greedy accuracy is computed from a separate deterministic response and is distinct from sampled Pass@1.Source: A EXPANDED FIXED-CONTEXT PASS@K ANALYSIS · p. 19

Results and robustness

B ADDITIONAL ANALYSIS OF VISUALLY SENSITIVE TOKENS

Teacher–student KL can reflect language variation rather than reliance on visual evidence. The visual-budget intervention more selectively identifies positions whose predictions respond to additional visual evidence.

  • High teacher–student KL can arise from language-level variation, including capitalization, paraphrases, equivalent wording, or response-format differences, without visual evidence driving the discrepancy.Source: B ADDITIONAL ANALYSIS OF VISUALLY… · p. 19
  • Some high-KL positions remain largely unchanged under the visual-budget intervention, indicating low visual sensitivity.Figure 6 provides examples of this pattern.Source: B ADDITIONAL ANALYSIS OF VISUALLY… · p. 19
  • The intervention-based sensitivity signal highlights positions whose predictions respond to additional visual evidence, better matching the goal of selecting useful privileged supervision than disagreement alone.Source: B ADDITIONAL ANALYSIS OF VISUALLY… · p. 19

C TRAINING AND EVALUATION DETAILS

The experiments adapt language models to sparse visual contexts using VisionZip, on-policy teacher supervision, and comparisons across pruning methods and image and video benchmarks.

  • VisionZip retains attention-dominant visual tokens and merges similar remaining tokens into contextual tokens; Qwen2.5-VL-7B-Instruct and VisionZip are the default model and pruning operator.The experiments also examine diversity-based, random, and in-LLM query-aware pruning, and transfer to Qwen3-VL-4B.Source: C TRAINING AND EVALUATION DETAILS · pp. 20–21
  • The retention ratio b is measured against the preprocessed visual sequence after a 1280-token cap; training uses 10%, while evaluation also tests larger budgets.VisionZip selects tokens independently of the question and generated text, leaving the retained visual context fixed during a reasoning trajectory.Source: C TRAINING AND EVALUATION DETAILS · p. 21
  • Post-training uses 10K LLaVA-CoT-derived examples, but SCOPD and SCOPD+ use only images and questions, obtaining online supervision from the full-context teacher rather than ground-truth answers or reasoning traces.Source: C TRAINING AND EVALUATION DETAILS · p. 21
  • The sparse-context student generates on-policy reasoning trajectories, and the full-context teacher scores the same student-generated prefixes.Training uses language-model LoRA while the vision encoder and multimodal projector remain frozen; SCOPD+ retains the top ρ = 0.1 fraction of response positions by visual sensitivity.Source: C TRAINING AND EVALUATION DETAILS · p. 21
  • Baselines include Vanilla, SFT, EPIC, and GRPO, while token-selection ablations test random selection, high-KL positions, TIP, and low-sensitivity positions.These comparisons probe whether SCOPD+ gains relate to sparse backpropagation, generic disagreement, or visual sensitivity.Source: C TRAINING AND EVALUATION DETAILS · pp. 21–22
  • Evaluation covers 13 image benchmarks and five video benchmarks, using greedy decoding and a shared reasoning prompt; ambiguous image-benchmark answers receive semantic answer matching.Video evaluation uniformly samples 32 frames and allows up to 12,288 visual tokens per video.Source: C TRAINING AND EVALUATION DETAILS · p. 22

D EXTREME COMPRESSION AT 5% VISUAL-TOKEN RETENTION

At 5% visual-token retention, SCOPD and SCOPD+ improve normalized Avg8 over VisionZip, though a gap to the unpruned model remains.

D EXTREME COMPRESSION AT 5% VISUAL-TOKEN RETENTION

E GRPO DISCUSSIONS

GRPO training leaves accuracy, Pass@4, and response length unchanged; most groups provide no gradient because all sampled responses are already correct.

E GRPO DISCUSSIONS

  • Accuracy, Pass@4, and response length do not change during a 2,500-step GRPO run, while format compliance improves slightly.The run takes about 30 hours; the setup samples 16 responses per step across four GPUs.Source: E GRPO DISCUSSIONS · p. 23
  • With four samples per prompt, over half of prompts have four correct responses, yielding zero-advantage groups that contribute no gradient.Source: E GRPO DISCUSSIONS · p. 23
  • Only roughly one third of groups have mixed outcomes, leaving too little learning signal to move the policy within the compute budget.Larger groups or harder prompts could increase informative groups but cost proportionally more per step.Source: E GRPO DISCUSSIONS · p. 23

F GROUND-TRUTH-AUGMENTED DISTILLATION

Default SCOPD uses on-policy student trajectories and full-context teacher supervision without ground-truth answers; an augmented variant supplies the teacher with answers when available.

F GROUND-TRUTH-AUGMENTED DISTILLATION

G COMPUTE

SCOPD+ adds training compute for its visual-sensitivity intervention, with a much smaller measured increase in wall-clock time and peak memory; neither method adds inference-time computation.

G COMPUTE

  • The visual-budget intervention and privileged teacher are removed at inference, and deployed models use the pruned VLM’s same architecture and sparse visual context.Source: G COMPUTE · p. 24
  • Mean generated tokens are 155.27 for Vanilla, 153.00 for SCOPD, and 155.70 for SCOPD+, so adaptation does not produce longer-than-base trajectories.With the LoRA setup, full post-training on four NVIDIA L40 GPUs takes approximately 14 hours.Source: G COMPUTE · p. 24
  • SCOPD+ uses 65.84 TFLOPs per sample versus SCOPD’s 54.25 TFLOPs, a 21.4% increase, while time rises 1.9% and peak memory 0.4%.On one NVIDIA L40 GPU, wall-clock time is 17.54 s versus 17.21 s per sample, and peak memory is 25.13 versus 25.02 GiB.Source: G COMPUTE · p. 24, Table 9 · p. 25
Loading 2609.34044v1…