PaperlayerView exact revision on arXiv

Source-linked AI summary

Aggregating Visual Information with Optimal Transport for VideoLM Token Compression

Wenti Yin, Xiaotian Han, Junyuan Shang, Yuchen Ding, Shuohuan Wang, Dianhai Yu, Changxin Gao, Nong Sang

arXiv:2608.20473v1cs.CV

TL;DR

VideoLMs face redundant, rapidly growing visual-token sequences, making compression important while preserving information distributed across frames. AVIOT uses optimal transport to construct compact representations, conditioning temporal allocation on questions and fusing region-specific spatial representations. Across multiple benchmarks and compression ratios, it matches or outperforms the uncompressed baseline while retaining strong higher-ratio performance.

  • Problem

    VideoLM visual-token sequences grow with sampled frames and contain substantial repetition, creating a compression challenge that must preserve information dispersed across frames.

  • Method

    AVIOT transports a dense empirical measure of frame observations onto a compact target measure, using question-conditioned transport and spatially granular plans to construct and fuse compressed representations.

  • Results

    AVIOT matches or surpasses the uncompressed backbone on multiple video-understanding benchmarks and remains competitive at higher compression ratios.

  • Takeaways & Limitations

    Optimal-transport representation construction preserves strong video-understanding performance while reducing the visual-token sequence passed to language-model decoding.

  • Takeaways & Limitations

    AVIOT reduces language-model prefill cost after frame encoding but does not reduce vision-encoder computation, and broader backbones, encoders, layouts, and streaming settings remain unevaluated.

Abstract

from arXiv · show

Video language models process videos as dense visual-token sequences with substantial representational redundancy. Compressing these sequences is therefore essential for reducing the visual-token burden on language-model decoding. The central challenge is to preserve visual information dispersed across frames under such compression. To this end, we introduce Aggregating Visual Information with Optimal Transport (AVIOT), which casts video token compression as transporting a dense empirical measure of frame observations onto a compact target measure. The resulting source-to-target coupling induces a distribution over source observations for each target support, directly specifying how the compressed video representation is constructed. We further adapt this construction along task and spatial axes. Question conditioning modulates the transport cost between source frames and target supports, while influencing how many supports are allocated to each temporal segment, thereby directing representation capacity toward question-relevant content. At multiple spatial granularities, AVIOT computes region-specific temporal transport plans and adaptively fuses the representations they yield, allowing different regions within the same compact representation to draw from different moments. Evaluations across varying compression ratios show that AVIOT matches or outperforms the uncompressed baseline on multiple video-understanding benchmarks while retaining strong performance at higher compression ratios.

1 Introduction

AVIOT treats video-token compression as constructing compact representations that jointly aggregate information dispersed across frames. It uses optimal-transport coupling, question-conditioned allocation, and spatially granular aggregation to preserve relevant visual structure under compression.

  • Motivation: VideoLM visual sequences grow with sampled frames because each frame contributes a spatial token grid, creating redundancy and increasing inference computation and memory.Slowly changing scenes, objects, and backgrounds contribute substantial repetition.
  • Existing compression paradigms: Existing approaches either select or merge source observations, or map dense features into compact learned bottlenecks.These routes reduce source observations directly or synthesize a small output set through learned feature mappings.
  • Optimal-transport representation construction: AVIOT formulates compression as transporting a dense empirical measure of frame observations onto a compact target measure with a prescribed number of supports.Each target support receives a normalized incoming source distribution that constructs its descriptor and full spatial features through barycentric projection.
  • Task-conditioned allocation: Question conditioning changes source-to-target transport costs and allocates supports across temporal segments using question relevance and preliminary transport responses.This directs representation capacity toward content associated with the question.
  • Evaluation: AVIOT matches or surpasses the uncompressed backbone on multiple benchmarks and remains competitive at higher compression ratios.The evaluation spans varying compression ratios and video-understanding benchmarks.
  • Spatially adaptive aggregation: Spatially granular transport produces global, medium, and local representations whose region-specific temporal correspondences are adaptively fused while preserving spatial layout.Different regions of one compact representation can therefore draw from different source moments.

4 Experiments

AVIOT is evaluated across diverse video-understanding benchmarks, compression ratios, ablations, and provenance analyses. It preserves strong performance under compression while question conditioning and spatially adaptive transport contribute complementary benefits.

  • Evaluation setup: AVIOT is evaluated on ten video-understanding benchmarks using a controlled 64-frame comparison and additional open-source VideoLM comparisons.The controlled benchmarks include Video-MME, EgoSchema, MVBench, ActivityNet-QA, and Perception Test; further evaluations cover five additional benchmarks.
  • Main results: At compression ratios 2 and 4, AVIOT exceeds the uncompressed backbone by 2.36 and 1.02 points on average, respectively.At ratio 2, it improves four of five benchmarks and remains within 0.08 points on Video-MME.
  • Main results: At ratio 10, AVIOT’s mean decrease is only 1.12 points, while ActivityNet-QA, Perception Test, and MVBench match or exceed the backbone.At this ratio, AVIOT also outperforms Uniform Keep and Segment Mean on every benchmark.
  • Compression-ratio generalization: AVIOT generalizes beyond its training range: the five-benchmark mean changes from 64.38 at r = 2 to 60.74 at r = 20.Doubling compression beyond the largest training ratio reduces the mean by 1.53 points without additional optimization.
  • Question conditioning: Question conditioning reallocates supports toward question-relevant temporal segments, assigning 33.0% to the final segment for the food-preparation question versus 11.4% for the spoon-color question.The total support budget and other experimental settings are held fixed.

5 Conclusion

The conclusion presents AVIOT as an optimal-transport framework for compact video representations. It emphasizes question-adaptive temporal construction, spatially region-specific aggregation, and strong performance across benchmarks and compression ratios.

  • Conclusion: AVIOT constructs compact video representations through a joint source-to-target coupling.Question conditioning adapts temporal correspondence and support allocation, while spatial-granularity transport enables region-specific aggregation.
  • Conclusion: Across ten benchmarks, AVIOT maintains strong performance across compression ratios, supporting optimal-transport representation construction for VideoLM token compression.

Supplementary Material

The supplementary material contains additional method details and evidence for AVIOT, including implementation, quantitative, qualitative, and visual-token analyses.

  • Supplementary Material: The supplementary material expands AVIOT’s representation construction, numerical realization, architecture, and training objective.It also presents extended quantitative, qualitative, and visual-token accounting analyses.

S1 Supplementary Overview

The supplementary overview identifies the material’s purpose and summarizes its methodological and analytical extensions.

  • Supplementary Overview: The supplementary material provides additional method details and evidence for AVIOT.Its contents include expanded representation construction, numerical realization, architecture, training objective, and extended analyses.

S2 Additional Method Details

AVIOT augments visual features with explicit spatiotemporal coordinates, then allocates and refines temporal supports using question-conditioned and spatially granular transport. Exact feature-mixture coefficients and adaptive fusion preserve provenance while combining information across regions and times.

  • Position-Aware Video Features: AVIOT adds Fourier-encoded time–height–width coordinates through a learned projection without changing the visual encoder output shape.The positional residual is introduced gradually during the first 500 optimization steps.
  • Question Conditioning and Temporal Allocation: Question conditioning influences transport costs, temporal support allocation, and all spatial granularities within a compression stage.The relevance path informs temporal allocation, while the transport cost is separately question-conditioned.
  • Question Conditioning and Temporal Allocation: Temporal supports are allocated across at most four contiguous segments using pilot transport responses and language-model-space relevance signals.Budgets are rounded, capped by available observations, repaired deterministically, and concatenated in temporal order.
  • Transport across Spatial Granularities: AVIOT computes global, medium, and local regional plans, with one 27 × 27 global region, nine 9 × 9 medium regions, and 81 3 × 3 local regions.Regional plans use aligned support budgets and are adaptively fused through nonnegative weights that sum to one.
  • Progressive Construction: AVIOT supports inference-time target cardinalities from progressive paths that include ratios 2, 4, 10, and 20 for T = 224.The corresponding paths end at 112, 56, 23, and 12 supports, respectively.
  • Exact Provenance: The final representation retains exact source-to-target feature-mixture coefficients produced by the executed construction rather than post-hoc attention or saliency estimates.Progressive stages compose coefficients relative to the original source video.

S3 Numerical OT Construction

The numerical construction uses finite-step, log-domain Sinkhorn updates to approximate entropy-regularized OT couplings and explicitly normalize target columns. One progressive stage allocates supports temporally, refines global plans, then derives aligned regional plans for fusion.

  • Finite-Step Log-Domain Updates: AVIOT approximates the entropy-regularized OT coupling with finite-step log-domain Sinkhorn updates.The resulting finite-step coupling is used directly for representation construction.
  • Target-Conditional Construction: Each target column is explicitly normalized into the target-conditional source distribution used to construct a compressed support.The damped updates approach the standard marginal maps before this normalization.
  • Support Refinement: AVIOT recomputes question-conditioned costs after support updates, using five global refinement rounds, two medium rounds, and one local round.The global, medium, and local entropy coefficients are 0.10, 0.12, and 0.15.
  • Boundary Case: Segments use an identity plan when target supports equal source observations, preserving the segment exactly without a numerical solve.This is a boundary case of the progressive construction.
  • Stage Algorithm: A progressive stage forms descriptors, splits observations into at most four contiguous segments, computes pilot allocations, and initializes supports uniformly in time.Global plans are refined within segments and concatenated in temporal order.
  • Regional Refinement: The stage then initializes and refines nine medium plans and 81 local plans from coarser representations before restoring spatial layout and fusing branches.Regional branches share segment boundaries and support budgets, keeping target indices aligned.

S4 Architecture and Training Details

AVIOT trains a progressive, multi-granularity compression architecture with distortion, regional continuity, fusion smoothness, and anti-collapse terms. It jointly optimizes the language model and compression components while freezing the visual encoder and sampling training ratios from 2 to 10.

  • Compression Objective: The compression objective combines global and regional coupling-weighted distortion with regional continuity, spatial fusion smoothness, and granularity-collapse penalties.The resulting objective is averaged across executed stages before being added to answer-generation loss.
  • Regularization: Regional continuity regularization encourages neighboring regional plans to vary smoothly while allowing the plans themselves to differ.The continuity term uses adjacent-region pairs and normalized regional descriptors.
  • Compression Objective: AVIOT averages the effective objective over J executed progressive stages before adding it to the answer-generation loss.This couples intermediate compression stages to the training objective.
  • Model Components and Training: The model initializes from LLaVA-Video-7B-Qwen2, freezes SigLIP, and jointly optimizes the language model, projector, positional projections, transport components, and fusion gate.Training uses LLaVA-Video-178K for 10,903 steps.
  • Training Schedule: Training samples uniformly from 17 compression ratios spanning 2 to 10 in increments of 0.5.At inference, the ratio is supplied externally and is not restricted to the training grid.

S5 Extended Quantitative Results

Across extended benchmark analyses, AVIOT remains effective as compression increases, with degradation varying by dataset and task type. It also benefits from denser temporal input and substantially reduces decoder-prefill cost while preserving broad scene and state evidence.

  • Controlled TempCompass Results: AVIOT achieves the highest score among three compressed methods on every controlled TempCompass task and ratio.Its task average is 67.80, 67.05, and 56.74 at r = 2, 4, and 10.
  • Performance Across Compression Ratios: At r = 10, AVIOT exceeds the published LLaVA-Video-7B reference on Video-MMMU, ActivityNet-QA, Perception Test, and MVBench.The reported pairs are 37.89 vs. 36.10, 66.26 vs. 64.10, 69.80 vs. 67.90, and 59.60 vs. 58.60, respectively.
  • Performance Across Compression Ratios: At r = 10, the ten benchmark scores retain 89.8–99.1% of their respective r = 2 values, and eight retain more than 93%.From r = 10 to r = 20, the additional decrease on five reported benchmarks is at most 2.22 points.
  • Robustness Across Input Sampling: Denser input sampling improves ActivityNet-QA, Perception Test, MVBench, Video-MME w/o sub., and EgoSchema relative to fixed 64-frame input at the displayed ratios.The gains are reported separately for ratios 2, 4, and 10, with Video-MME w/o sub. gains of 1.89, 3.04, and 3.69 points.
  • MVBench Task Breakdown: Increasing MVBench compression from r = 2 to r = 10 keeps 13 of 20 tasks within five points and 19 within eight points of the r = 2 accuracy.Broad event and state evidence remains stable, while fine temporal distinctions change more with compression.
  • Visual-Token and Decoder-Prefill Accounting: At r = 2, 4, and 10, standardized decoder-prefill counts fall by 58.5%, 81.4%, and 92.4% relative to the 64-frame visual prefix.At fixed target cardinality, AVIOT and the comparison methods present the same token layout but construct supports differently.

S7 Additional Qualitative Analysis

AVIOT provides controlled and qualitative evidence that question-conditioned allocation and multi-granularity spatial transport adapt compact representations to relevant temporal and spatial content. Provenance analyses show that compressed supports combine localized, transitional, and repeated interactions while retaining broader context.

  • Exact Provenance Across Video Types: Exact feature-mixture coefficients reconstruct each compressed feature up to numerical precision and are not post-hoc attention or saliency estimates.
  • Question-Conditioned Temporal Allocation: Matched questions allocate support capacity differently across temporal segments, emphasizing spoon and mixing for one question and the food-revealing segment for another.The allocations are [23, 25, 30, 10] and [20, 20, 19, 29], respectively.
  • Question-Conditioned Temporal Allocation: 21.6% of the stage budget is reallocated between segments, with a temporal divergence of 0.155 bits under matched question conditioning.
  • Question-Conditioned Temporal Allocation: The controls attribute allocation changes to the compressor query: rotated queries nearly remove the difference, while no query produces identical allocations and feature-mixture tensors.Rotated queries reduce allocation distance to 0.023 and temporal divergence to 0.022 bits; no compressor query makes both measures zero.
  • Transport Across Spatial Granularities: For a fine object-state transition, global, medium, and local paths contribute 23.6%, 16.8%, and 59.6% of one fused provenance mass.The global and medium paths jointly retain 40.4%, while branch maps focus on the same interaction but differ spatially.
  • Exact Provenance Across Video Types: Provenance cases cover localized, transitional, and repeated interactions while combining temporal development, actor–object evidence, and broader scene context.Examples include installation views, a liquid-to-solid transition, and repeated knife-sharpening states.

S8 Limitations

AVIOT reduces visual tokens passed to the language model and therefore lowers language-model prefill cost, but its scope excludes vision-encoder computation and several broader deployment settings.

  • AVIOT is applied after sampled frames are encoded, so it does not reduce vision-encoder computation.
  • Experiments use LLaVA-Video-7B-Qwen2, a regular 27×27 SigLIP grid, and non-streaming video, leaving other backbones, encoders, layouts, and streaming settings for future work.
  • The compression ratio is specified at inference time, while automatic video- and question-dependent ratio selection remains future work.

Liquid-to-Solid Egg Transition | Frame- and Token-Level Provenance

This provenance example examines a liquid-to-solid egg transition in a 121-frame video compressed to 16 supports, alongside frame- and token-level views for interpreting the support’s source evidence.

  • The example asks what state change occurs when yellow egg mixture is poured into a water-filled pot.
  • The displayed provenance example uses final support 4 from 121 source frames compressed to 16 supports at source frame 4, or 2.00 seconds.
  • The frame-level curve reports the support’s source-frame weights across the displayed interval, while token-level views show its spatial construction within the displayed frame.
  • The accompanying knife-sharpening panel provides a separate fine-grained tool-interaction provenance example with 77 frames compressed to 10 supports.
Loading 2608.20473v1…