Source-linked AI summary
Compressing AI Traffic: Standardized Neural Network Coding of Visual-Token Representations in Split Vision-Language Inference
Reza Heidari, Hamed R. Tavakoli, Juho Kannala
TL;DR
Split VLM deployments communicate intermediate visual representations, creating a need to reduce AI traffic for bandwidth, latency, and system efficiency. The paper applies standardized Neural Network Coding to the complete visual interface of Qwen3-VL and finds that downstream performance remains near baseline under extreme compression before collapsing, despite distorted reconstructions.
Problem
Split VLM deployments transmit intermediate visual representations between separate compute nodes, motivating compression to reduce bandwidth, latency, and system cost.
Method
The study independently applies standardized ISO/IEC 15938-17 Neural Network Coding to Qwen3-VL's primary visual-token representation and DeepStack feature streams before language-model inference.
Results
Up to 98% reduction relative to the BF16 source tensor preserves essentially unchanged downstream task performance, after which both benchmarks show sharp collapse.
Takeaways & Limitations
Strong compression tolerance despite discretized and distorted reconstructions supports evaluating AI traffic codecs with rate–task rather than rate–distortion curves.
Takeaways & Limitations
The study covers one backbone and subsets of two benchmarks, while open-ended evaluation relies on an LLM judge.
Abstract
from arXiv · showhide
When the visual encoder and the language decoder of a vision-language model (VLM) run on different compute nodes, the intermediate visual-token embeddings become a communicated payload rather than an internal activation. We call such machine-consumed intermediate tensors AI traffic and ask how far they can be compressed with a standardized, training-free codec. We insert ISO/IEC 15938-17 Neural Network Coding (NNC) round trips on the complete visual interface of a Qwen3-VL-8B-Instruct video question answering pipeline, comprising the main visual-token representation and the DeepStack feature streams, while leaving weights, prompts, and generation untouched, and sweep the quantization parameter (QP) over a wide rate range. Closed-ended Video-MME accuracy remains close to the uncompressed reference up to a 98% reduction of the transmitted BF16 tensor and only then collapses; open-ended MLVU generation shows the same plateau-and-collapse profile under an LLM judge. This robustness is not due to near-lossless reconstruction: the decoded tensor is heavily discretized, carries substantial row-wise relative L2 error, and has a visibly steeper singular-value decay than its source. Downstream reasoning therefore depends on coarse structure and relative geometry rather than exact floating-point values, which argues for rate-task rather than rate-distortion optimization of AI traffic codecs.
I. INTRODUCTION
Split VLM deployments turn intermediate visual tokens into bandwidth-relevant AI traffic. The paper applies standardized NNC compression to the complete visual interface and finds substantial compression tolerance before downstream performance collapses.
- I. INTRODUCTION: Video makes the visual interface expensive because long token sequences encode spatial content, temporal change, event ordering, and long-range context.
- I. INTRODUCTION: Split deployments can transmit intermediate visual tokens instead of raw video, making them AI traffic exchanged between encoder and decoder.Reducing this traffic matters for bandwidth, latency, and system efficiency.
- I. INTRODUCTION: The study applies ISO/IEC 15938-17 Neural Network Coding to Qwen3-VL’s primary visual tokens and DeepStack feature streams before language-model inference.Each stream is independently encoded, decoded, and replaced by its reconstruction.
- I. INTRODUCTION: Very aggressive rate reduction is tolerated before performance collapses, even though reconstructions are strongly discretized and substantially distorted.The decoder can operate when enough task-relevant structure survives rather than requiring exact floating-point reconstruction.
- I. INTRODUCTION: The paper evaluates standardized, training-free visual-token transport on closed-ended and open-ended video understanding, including identity and zero-token controls.
II. RELATED WORK
The paper places visual-token compression within neural-network and machine-oriented coding research. Its distinction is a fixed standardized tensor coder for a generative VLM interface rather than a task-trained compressor for conventional feature maps.
- II. RELATED WORK: Qwen3-VL-8B-Instruct is a large VLM with long interleaved contexts and video-specific spatio-temporal modeling and temporal grounding.
- II. RELATED WORK: Figure 1 shows the main visual-token representation and DeepStack streams independently NNC encoded, transmitted, decoded, and substituted before language-model inference.
- II. RELATED WORK: NNC provides a compressed representation and decoding process for neural-network data, using DeepCABAC entropy coding and the NNCodec reference software.
- II. RELATED WORK: Unlike prior collaborative-intelligence work that generally trains compression for a downstream network, this study uses a fixed, standardized, training-free coder for a generative VLM interface.
III. COMPRESSING AI TRAFFIC
AI traffic compression is evaluated by whether a receiving model retains task-relevant information, not by whether the decoded representation resembles the original numerically or visually.
- III. COMPRESSING AI TRAFFIC: AI traffic targets machine reasoning, so decoded representations need not be visually meaningful if they preserve what the receiving model requires.
- III. COMPRESSING AI TRAFFIC: The visual interface consists of a primary representation and K DeepStack streams, with K = 3 in Qwen3-VL-8B-Instruct.
- III. COMPRESSING AI TRAFFIC: Each interface tensor is independently coded using the same NNC operating point, and the reconstructed tensors replace the originals before language-model inference continues.QP controls compression strength and is shared across all streams.
- III. COMPRESSING AI TRAFFIC: Downstream task performance is the primary useful-reconstruction measure because large numerical error may coexist with preserved decoder structure, while numerical accuracy may not preserve task-relevant relations.
IV. EXPERIMENTAL SETUP
The experiment inserts NNC round trips into the complete Qwen3-VL visual interface while holding the model pipeline fixed, then evaluates task performance across compression conditions.
- IV. EXPERIMENTAL SETUP: The visual interface comprises one primary visual-token representation and three DeepStack streams, each independently encoded and decoded before language-model inference.Weights, tokenizer, prompts, decoder, and generation remain otherwise unmodified.
- IV. EXPERIMENTAL SETUP: Only QP varies across the NNC sweep; tensors are prepared in FP32 for encoding and decoded back to the pipeline’s expected device and dtype.
- IV. EXPERIMENTAL SETUP: Identity provides the uncompressed reference, zero-token replaces the visual interface with zeros, and NNC supplies the reconstruction condition.
- IV. EXPERIMENTAL SETUP: Video-MME measures strict option-letter accuracy and valid rate, while MLVU summary generations receive GPT-4 completeness and reliability scores summed to 10.Each benchmark uses 100 deterministically sorted test samples, and compression is measured against the aggregate BF16 size of the complete visual interface.
V. RESULTS
NNC compression preserves Video-MME and MLVU performance across a broad rate range, but aggressive compression produces a sharp collapse. Video-MME remains near the identity reference at roughly 99% BF16 reduction before performance degrades.
- Compression relative to the BF16 source spans 58.4% at QP = −32 to 99.99% at QP = 12.Increasing QP monotonically shrinks the encoded bitstream.
- Across low to fairly high QP, Video-MME accuracy tracks the identity baseline before falling under aggressive compression.The zero-token baseline provides a visual-information sanity check.
- Video-MME accuracy stays within a few points of the reference at QP = 0, where the bitstream is around 99% smaller than the BF16 tensor.The zero-token condition drops accuracy from 0.74 to 0.21, while valid parsing remains perfect throughout the plateau.
- MLVU shows the same pattern: moderate compression preserves most judged generation quality, whereas aggressive compression degrades it.This extends the plateau-and-collapse behavior beyond multiple-choice answering to free-form output.
- Both benchmarks exhibit a plateau-and-collapse profile, with performance near the identity baseline until a small additional rate reduction causes a sharp drop.The useful operating region reduces communication cost by one to two orders of magnitude at essentially no task cost.
VI. REPRESENTATION-LEVEL ANALYSIS
NNC compression substantially distorts and discretizes the primary visual-token representation, yet downstream performance remains stable across the plateau region. The reconstructed representation also exhibits steeper singular-value decay, indicating reduced effective dimensionality.
- Substantial row-wise relative L2 error shows that NNC reconstruction is not near-identical to the source representation.The distortion is especially pronounced at stronger compression settings.
- Continuous source values collapse onto discrete reconstruction levels, revealing strong quantization.The reconstruction concentrates on a smaller set of discrete values than the comparatively smooth source distribution.
- Downstream performance remains stable over a wide QP range despite substantial value-level distortion.This stability shows that near-lossless reconstruction is not required for the evaluated tasks.
- The reconstruction’s singular values decay more steeply than the source’s, consistent with reduced effective rank.Compression suppresses or removes lower-energy directions, many of which appear non-essential in the plateau region.
VII. DISCUSSION AND CONCLUSION
The study concludes that visual-token AI traffic contains substantial compressible redundancy while retaining task-relevant information that cannot be discarded indefinitely. Standardized NNC enables large communication reductions without architectural changes or retraining, but the findings remain limited in scope.
- Compressing the visual-token interface reduces communication cost without architectural changes or retraining.
- NNC provides a standardized tensor-coding pipeline rather than an ad hoc quantizer, supporting interoperability between components from different vendors.
- The study is limited to one backbone and subsets of two benchmarks, so behavior may depend on architecture, tokenization, task, and interception point.Open-ended evaluation also relies on an LLM judge.
- Up to 98% reduction relative to the BF16 source tensor preserves essentially no loss in downstream task performance before sharp collapse.The result is reported for the tested split vision–language pipeline across its evaluated operating range.