PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Reason with Compressed Context: Ground-Truth-Free Adaptation of OmniLLMs via Self-Distillation

Jianghao Wang, Ke Meng, Jian Li, Chi Cheng, Longyu Qi, Liyin Liang, Yifeng Qian, Chunbo Lai, Yutian Lin, Zeyu Wang

arXiv:2609.39953v2cs.CV

TL;DR

Long multimodal token sequences make OmniLLM deployment costly, while compression can reduce accuracy and adapting the model to use compressed context is underexplored. CAFD distills from a full-context self-teacher to a compressed-context student without reference answers; it improves 120 of 125 evaluated conditions by 1.44 average accuracy points.

  • Problem

    Compression reduces deployment cost but can remove task-relevant evidence, and adapting an OmniLLM to a fixed compressor rather than redesigning that compressor remains less explored.

    Source: §1 · p. 1

  • Method

    CAFD uses a full-context self-teacher to provide soft targets along a compressed-context student's on-policy response trajectory, without reference answers, rationales, evidence annotations, or correctness rewards.

    Source: §1 · pp. 2–3

  • Results

    CAFD improves 120 of 125 matched conditions, raising average accuracy from 49.26% to 50.70% (+1.44 points) and recovering 26.9% of the accuracy gap on average.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Across the evaluated pipelines, benchmarks, and budgets, adapting the model complements compression, and a model adapted once at 5% retention can be reused across five deployment budgets.

    Source: §1 · p. 3

    Training uses a small dataset and a fixed retention budget for each model, leaving larger datasets and mixed-retention training for future investigation.

    Source: §5 · p. 9

Abstract

from arXiv · show

Omni-modal large language models (OmniLLMs) enable unified audio-video understanding, but their long multimodal token sequences make deployment computationally expensive. Token compression reduces this cost, yet aggressive compression often lowers accuracy. Existing works predominantly focus on designing better compression mechanisms; however, adapting the underlying language model to reason effectively over the remaining compressed context remains under-explored. To address this, we propose CAFD (Compressed-Context Adaptation via Full-Context Distillation), a ground-truth-free self-distillation framework that adapts OmniLLMs to fixed compression pipelines without requiring reference answers, rationales, or correctness rewards. CAFD leverages the full-token view of the same multimodal sample as a source of privileged information: a full-context self-teacher provides soft target supervision to a compressed-context student along the student's on-policy trajectory. Evaluated on Qwen2.5-Omni-7B across five audio-video benchmarks, five compression pipelines, and five deployment budgets, CAFD demonstrates consistent gains, improving 120 out of 125 conditions with an average accuracy boost of 1.44 points and recovering 26.9% of the accuracy gap on average. These results demonstrate that the proposed ground-truth-free adaptation offers an effective and practical route to improving the accuracy-efficiency trade-off in deployed OmniLLMs.

1 INTRODUCTION

Long audio-video inputs create costly multimodal token sequences, and compression can remove task-relevant evidence. CAFD adapts an OmniLLM to use compressed context without changing the compressor, improving accuracy across the evaluated conditions.

1 INTRODUCTION

  • Long-form audio and video can produce thousands of multimodal tokens, increasing prefill computation and memory costs and motivating token compression for deployment.Existing methods prune, merge, select, or progressively reduce tokens before or within the language model.Source: §1 · p. 1
  • Compression can remove or weaken task-relevant evidence, while prior work chiefly improves compressors, leaving adaptation to a fixed pipeline less explored.A deployed system may already have a chosen compressor, so the model's ability to use the remaining information is a complementary focus.Source: §1 · p. 1
  • CAFD uses a full-context self-teacher to provide soft targets along a compressed-context student's own response trajectory, without reference answers, rationales, evidence annotations, or correctness rewards.The teacher uses the same sample's full-token view and is discarded at deployment; the compressor and inference architecture remain unchanged.Source: §1 · pp. 2–3
  • 120 of 125 matched conditions improve, with average accuracy rising from 49.26% to 50.70% (+1.44 points) and mean accuracy-gap recovery of 26.9%.At nominal 5% retention, the average gain is 1.73 points; the adapted model is reused across five deployment budgets.Source: Figure 1 · p. 2, §1 · p. 3

2 RELATED WORK

Prior work addresses token compression, on-policy self-distillation, and adaptation to compressed inputs through varied mechanisms and supervision. CAFD complements these approaches by adapting the OmniLLM under a fixed audio-video compression pipeline without reference answers or localized evidence.

2.1 OMNILLM TOKEN COMPRESSION

  • Compression methods reduce encoder outputs, intermediate decoder states, or both, using token saliency, redundancy, cross-modal cues, query relevance, or compact memory tokens.Some compressors are training-free, while others use profiling or train auxiliary or joint compression modules; CAFD instead adapts the OmniLLM under a fixed pipeline.Source: §2.1 · p. 3

2.2 ON-POLICY SELF-DISTILLATION

2.3 ADAPTATION TO COMPRESSED CONTEXTS

  • Compressed-context adaptation includes supervised tuning, full-to-compressed matching, teacher-forced distillation, and on-policy distillation, while ground-truth-free on-policy adaptation across audio-video compression pipelines remains underexplored.CAFD reports gains across token deletion, merging, and inner-LLM pruning.Source: §2.3 · p. 4

3 METHOD

CAFD adapts an OmniLLM to compressed audio-video inputs through self-distillation while leaving the compression pipeline unchanged. A full-context self-teacher supervises the compressed student on the student's sampled response trajectory.

3.1 PROBLEM FORMULATION

  • CAFD adapts the model under a selected compressed-context path without redesigning the compressor, covering pre-LLM, inner-LLM, and hybrid compression.The sample is paired with a textual instruction, and optimization updates trainable model parameters under the chosen fine-tuning parameterization.Source: §3.1 · p. 5

3.2 FULL-CONTEXT SELF-TEACHER

  • The full-context self-teacher processes the same sample with EMA parameters and supplies full-vocabulary supervision, while the compressed student uses the deployment context; neither path uses reference answers or rationales.The teacher also receives no evidence annotation or correctness-derived reward.Source: §3.2 · p. 5

3.3 ON-POLICY SELF-DISTILLATION OBJECTIVE

  • Because compression can change student responses, CAFD samples an on-policy trajectory from the current compressed student and computes teacher and student next-token distributions along its prefixes.The student distributions use the decoding settings; the teacher and student distributions are computed at the student-generated prefixes.Source: §3.3 · p. 5
  • CAFD matches the distributions with Jensen–Shannon divergence and pointwise clipping, averaging loss over generated response positions while gradients flow only through student distributions.After each successful optimizer step, teacher parameters track the student through an exponential moving average; deployment discards the teacher and retains the original compressor and inference architecture.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Experiments evaluate CAFD across benchmarks, compression pipelines, and deployment budgets, finding broad accuracy gains and benefits from both the distillation objective and ground-truth-free supervision. A controlled comparison also shows that CAFD can preserve average accuracy at lower nominal retention.

4.1 EXPERIMENTAL SETTINGS

  • Evaluation uses Qwen2.5-Omni-7B on five audio-video benchmarks and five pipelines spanning token selection, merging, and progressive inner-LLM pruning.Training uses 5,700 instances from 1,485 videos without reference answers or rationales; each compressor-specific model is trained at 5% retention and reused across five deployment budgets.Source: §4.1 · p. 6, §4 · p. 6

4.2 MAIN RESULTS

  • CAFD improves 120 of 125 matched conditions, raising average accuracy from 49.26% to 50.70% (+1.44 points) and recovering 26.9% of the accuracy gap.At nominal 5% evaluation retention, average accuracy rises from 44.31% to 46.04% (+1.73 points), with gains in all 25 conditions.Source: §4.2 · p. 7, Table 1 · p. 7
  • Adapted DivPrune at 25% retention reaches 53.01% accuracy versus 52.86% for unadapted DivPrune at 35%, while adapted DivPrune-Merge at 15% reaches 52.13% versus 52.03% at 25%.These comparisons show average accuracy can be preserved or improved at lower nominal retention.Source: §4.2 · p. 8

4.3 ADDITIONAL EXPERIMENTS

  • JSD outperforms forward and reverse KL on all three evaluated benchmarks at every tested retention budget.The objective comparison uses DivPrune at 5% training retention, with accuracy averaged over five deployment budgets within each benchmark.Source: §4.3 · p. 9, Table 2 · p. 9
  • CAFD achieves comparable performance without reference answers or rationales, while supplying references to the teacher yields no three-benchmark average-accuracy improvement over CAFD.Compressed-input SFT attains the highest average accuracy but uses reference responses as training targets.Source: §4.3 · p. 9, Table 3 · p. 9

5 LIMITATIONS AND FUTURE WORK

The current training setup is constrained by computing resources, using a small dataset and a fixed retention budget per model; future work could test larger datasets and mixed-retention training.

5 LIMITATIONS AND FUTURE WORK

  • Limited computing resources constrain training to a small dataset and a fixed retention budget per model.The authors propose investigating larger datasets and mixed-retention training for robustness across deployment budgets.Source: §5 · p. 9
  • Future work could investigate how teacher reliability and retained evidence inform distillation-objective design, and jointly optimize compression and adaptation under a fixed compute budget.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents CAFD as ground-truth-free model adaptation for reasoning over compressed multimodal context, using full-token inference to supervise a compressed-context student. Across five pipelines, five benchmarks, and five budgets, it reports recovery of a substantial portion of compression-related accuracy loss and generalization from one adaptation budget to others.

6 CONCLUSION

  • CAFD uses full-token inference as privileged information, distilling soft supervision from a full-context self-teacher to a compressed-context student along the student's response trajectory.The framework adapts the model without modifying the compression pipeline or requiring reference responses.Source: §6 · pp. 9–10
  • Across five compression pipelines, five audio-video benchmarks, and five deployment budgets, CAFD recovers a substantial portion of accuracy lost to compression and generalizes across deployment budgets from one adaptation budget.The conclusion frames efficient multimodal inference as both retaining useful tokens and learning to reason with the information that remains.Source: §6 · pp. 9–10

AI USE STATEMENT

The authors report using generative AI tools across literature review, research and experimental work, interpretation, manuscript preparation, and figure preparation, and state that they reviewed the work and checked numerical results.

AI USE STATEMENT

  • Generative AI tools assisted with literature review, hypothesis refinement, experimental design, implementation, interpretation, drafting, editing, and figure preparation; the authors reviewed the work and checked numerical results.The authors take responsibility for the paper's content and claims.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement directs readers to the paper’s framework, data and preparation, compressor settings, training details, and evaluation materials.

REPRODUCIBILITY STATEMENT

  • Section 3 gives the adaptation framework and objective, while Appendices B and C document implementation, data, and detailed results.Appendix B covers datasets, data preparation, compressors, budgets, prompts, hyperparameters, and evaluation protocols; Appendix C adds experimental results.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 TASK CONDITIONING AND SUPERVISION

The appendix tests reference-free task conditioning, video-disjoint in-domain adaptation, and training-seed sensitivity. Its reported results show gains without task-specific annotations and low variation across three seeds.

A.1 TASK CONDITIONING AND SUPERVISION

  • All three reference-free CAFD variants improve over Unadapted, and the fixed-generic variant uses media and a shared instruction without per-example QA fields.Supplying references to the teacher does not improve average accuracy over the main CAFD configuration; compressed-input SFT is strongest overall.Source: §A.1 · p. 14, Table 4 · p. 15

A.2 VIDEO-DISJOINT TARGET-DOMAIN ADAPTATION

  • Video-disjoint fixed-generic adaptation raises held-out WorldSense average accuracy from 43.08% to 44.00%, versus 43.73% for the OmniVideo-trained control.Evaluation uses 981 held-out QA instances from videos disjoint from adaptation data.Source: §A.2 · p. 15, Table 5 · p. 15

A.3 TRAINING-SEED SENSITIVITY

  • 53.57 ± 0.09% mean accuracy across three DivPrune training seeds indicates low variation in the reported runs.Runs used 5% retention and were averaged across three benchmarks and five deployment budgets.Source: §A.3 · p. 15

B.1 DATASETS AND STATISTICS

The appendices specify the datasets and controls, compression pipelines and budgets, training configurations, evaluation protocol, and comparison and aggregation rules used in the experiments.

B.1 DATASETS AND STATISTICS

  • The training subset contains 5,700 OmniVideo-100K QA pairs from 1,485 videos; evaluation spans five audio-video benchmarks with distinct task and media-duration coverage.The training prompt may use questions and options, but prescribed answers and rationales are not optimization targets.Source: §B.1 · pp. 15–16

B.2 DATA PREPARATION

  • Training videos are split from validation videos, prompt controls vary per-example QA information, and overlap checks found no shared evaluation items under the tested identifiers and fingerprints.The fixed-generic variant retains media paths and IDs but omits per-example QA fields.Source: §B.2 · p. 17

B.3 COMPRESSOR IMPLEMENTATIONS

  • The evaluated compressors implement distinct token-reduction strategies, including segment-midpoint sampling, modality-specific diversity selection, token merging, and joint-budget allocation.SEATS prunes before and within the LLM, so its reported retention is a layer-average estimate.Source: §B.3 · p. 17

B.4 COMPRESSION-BUDGET CONFIGURATIONS

  • Compression settings are fixed across datasets within each compressor and nominal budget; the tables distinguish joint retention targets from modality-specific or layer-average targets.Table 1 uses 5% for training and 5%, 10%, 15%, 25%, and 35% for evaluation.Source: §B.4 · p. 17, Table 8 · p. 18

B.5 TRAINING PROMPTS

  • Reference-free CAFD paths use the same textual prompt and differ in media-token views, while CAFD + reference adds teacher-only reference information and SFT trains on reference responses.Training prompts request concise analysis followed by one final option label.Source: §B.5 · pp. 17, 19, 20

B.6 TRAINING CONFIGURATION

  • Training updates only Thinker-decoder LoRA parameters, merges them before evaluation, and uses the specified optimizer, rollout, distillation, and hardware settings.The 5% setting was selected using four benchmarks; OmniVideoBench and Video-MME were not used for that selection.Source: §B.6 · p. 20

B.7 EVALUATION PROTOCOL

  • Evaluation uses audio-video inputs without subtitles, greedy decoding, and accuracy based on whether the initial generated option letter matches the correct label.Audio is limited to the first 300 seconds for OmniVideoBench and Video-MME, without truncating video.Source: §B.7 · p. 21, Table 10 · p. 21

B.8 COMPARISON SETUP

  • Each compressor-matched Unadapted/+CAFD pair differs only in Thinker weights and is evaluated across five budgets without retraining; benchmark and budget averages use equal weighting.Nominal retention labels can differ from realized retention, and accuracy-gap recovery is normalized by the full-token-to-Unadapted gap.Source: §B.8 · pp. 21–22

C.1 DETAILED COMPRESSION RESULTS

Across the reported behavioral analyses, adaptation yields more corrected than regressed predictions and improves correct-option scores, while detailed studies describe compute accounting, training choices, and task-conditioning effects.

C.1 DETAILED COMPRESSION RESULTS

  • Across fixed configurations, all 100 pre-LLM comparisons have identical token counts and analytical prefill computation, while 20 of 25 SEATS comparisons show small layer-computation changes.The prefill proxy counts listed matrix multiplications but excludes modality encoders, compression operations, and autoregressive decoding; unchanged SEATS retention estimates do not establish identical realized retention at every layer.Source: §C.1 · pp. 22–23, Table 11 · p. 23

C.2 TRAINING DATA SCALE AND RETENTION

  • The WorldSense data-scale subsets contain 570, 1,710, 2,850, and 5,700 instances, changing both distinct examples and optimizer updates; cross-compressor comparisons are not equal-compute comparisons.The corresponding update counts are 36, 108, 180, and 358.Source: §C.2 · pp. 25–26, Table 12 · p. 25

C.3 DISTILLATION OBJECTIVE

  • The objective study holds training configuration and the 0.05 pointwise clipping threshold fixed while varying vocabulary-level divergence among forward KL, reverse KL, and JSD.The runs use DivPrune at 5% training retention and evaluate the five listed deployment budgets.Source: §C.3 · p. 27, Table 14 · p. 27

C.4 TASK CONDITIONING AND SUPERVISION

  • Question-only CAFD slightly outperforms the main configuration on average, so including candidate options during adaptation does not necessarily improve accuracy.The expanded comparison covers five DivPrune deployment budgets across WorldSense, DailyOmni, and AVUT.Source: §C.4 · p. 28, Table 15 · p. 28

C.5 TRAINING DYNAMICS

  • Figure 4 tracks clipped-JSD training loss and student rollout length over training for DivPrune, OmniZip*, and SEATS.The plotted curves are 15-step trailing averages; loss contributions above 0.05 are capped and have zero local gradient through clipping.Source: Figure 4 · p. 28, §C.5 · p. 28

D.1 REPAIR AND REGRESSION

  • Corrected predictions outnumber newly incorrect predictions across three benchmarks and five DivPrune budgets, producing positive accuracy gains in every condition.DailyOmni shows the largest net accuracy gains across all five budgets.Source: §D.1 · p. 29, Table 16 · p. 29

D.2 CONFIDENCE AND RECOVERY DIAGNOSTICS

  • Correct-option probabilities, log-probabilities, and margins all increase at every budget on all three benchmarks, including cases where the final answer is unchanged.The reported recovery measures distinguish repairs within full-token-correct/compressed-incorrect cases from the net accuracy gain as a fraction of the overall gap.Source: §D.2 · p. 30, Table 17 · p. 30

D.3 RESULTS BY TASK TYPE

Task-category results vary across benchmarks: DailyOmni and AVUT gain accuracy in every listed category, while WorldSense has mixed changes. The largest WorldSense gains occur in attribute reasoning, event recognition, and video-emotion tasks.

D.3 RESULTS BY TASK TYPE

  • DailyOmni and AVUT show positive average accuracy gains in every listed task category, while WorldSense has larger gains in attribute reasoning, event recognition, and video-emotion tasks alongside unchanged or lower accuracy elsewhere.The table averages category metrics computed separately at each of five deployment budgets.Source: §D.3 · pp. 30–31, Table 18 · p. 31

Related papers

Loading 2609.39953v2…