The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems

Xiaoze Liu; Ruowang Zhang; Weichen Yu; Siheng Xiong; Liu He; Feijie Wu; Hoin Jung; Matt Fredrikson; Xiaoqian Wang; Jing Gao

Source-linked AI summary

The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems

Xiaoze Liu, Ruowang Zhang, Weichen Yu, Siheng Xiong, Liu He, Feijie Wu, Hoin Jung, Matt Fredrikson, Xiaoqian Wang, Jing Gao

arXiv:2602.15382v2Feb 17, 2026cs.CLcs.CVcs.LG

TL;DR

Heterogeneous MAS needs a way to avoid text communication overhead without assuming shared model architectures or maintaining pairwise latent translators. Vision Wormhole uses VLM visual pathways and universal codecs to transfer reasoning traces across model families, reducing runtime across most settings while producing positive macro-average Δ-accuracy.

Problem
Text communication imposes runtime overhead and quantization error, while latent transfer is limited by homogeneous architectures, disjoint manifolds, and pair-specific translators.
Method
Vision Wormhole maps reasoning traces into a shared continuous reference space with per-family codecs, injects decoded tokens into the receiver’s visual pathway, and trains through label-free distillation from the text channel.
Results
Vision Wormhole reduces end-to-end wall-clock time across most evaluated heterogeneous settings and yields positive macro-average Δ-accuracy, with gains concentrated in code generation.
Takeaways & Limitations
The framework provides a bounded, modular alternative to text communication and pairwise cache translators for heterogeneous VLM-based MAS.
Takeaways & Limitations
The reported results characterize a research-system setting using VLM-based agents, publicly released checkpoints, and the benchmark and protocol suite evaluated in the paper.

Abstract

from arXiv · show

Multi-Agent Systems (MAS) powered by Large Language Models have unlocked advanced collaborative reasoning, yet they remain bottlenecked by discrete text communication, which imposes runtime overhead and information quantization loss. While latent state transfer offers an alternative, existing approaches either assume homogeneous sender--receiver architectures or rely on pair-specific learned translators, limiting scalability across diverse model families with disjoint manifolds. We reconceptualize the visual interface of Vision-Language Models (VLMs), trained for natural images, as a continuous communication channel between heterogeneous agents, and instantiate this idea as the \textbf{Vision Wormhole}: a Universal Visual Codec maps reasoning traces into a shared continuous reference space and injects them into the receiver's visual pathway, yielding cross-architecture latent state transfer without per-pair translators. The framework adopts a hub-and-spoke topology that reduces alignment complexity from $O(N^2)$ to $O(N)$, and is trained by label-free teacher--student distillation against the text channel, requiring no parallel hidden-state supervision. Extensive experiments across heterogeneous VLM families (Qwen-VL, Gemma, SmolVLM2, LFM2.5-VL) and nine reasoning benchmarks show that the Vision Wormhole reduces end-to-end wall-clock time across most evaluated settings and yields positive macro-average $Δ$-accuracy.

1 Introduction

The paper addresses the runtime and interoperability limits of text and pairwise latent communication in heterogeneous MAS by repurposing VLM vision interfaces as a shared continuous channel. Its Vision Wormhole uses universal codecs, label-free distillation, and hub-and-spoke alignment to support scalable cross-family communication.

Motivation: Text communication adds runtime overhead and quantization error, while existing latent methods mainly target homogeneous models or require pairwise translators.These constraints limit heterogeneous MAS from combining diverse model strengths.
Challenges: Heterogeneous models occupy disjoint latent manifolds, making naive alignment difficult and simple linear mappings vulnerable to the off-manifold problem.The incompatibility includes differing dimensions and semantic geometries.
Approach: The Vision Wormhole repurposes VLM vision-token inputs to transmit dense reasoning information across disjoint model families without fine-tuning backbone parameters.The image soft embedding provides a fixed-length continuous pathway for meaningful context.
Approach: Pairwise translation requires quadratic O(N^2) alignment, whereas the proposed hub-and-spoke universal space reduces integration complexity to O(N).A new VLM family trains one codec rather than N pairwise translators.
Training: Label-free distillation uses the text channel as teacher and the vision wormhole as student, avoiding parallel hidden-state supervision.The latent channel learns task behavior through distribution and representation matching.
Evaluation: Across four VLM families and nine reasoning benchmarks, the method reduces end-to-end wall-clock time and yields macro-positive Δ-accuracy, with gains concentrated in code generation.The reported gain concentration includes MBPP-Plus and HumanEval-Plus.

2 Related Work

Prior work reduces communication or deliberation costs but remains limited by shared-backbone assumptions, pair-specific bridges, or single-model scope. Vision Wormhole instead preserves heterogeneous MAS workflows while using a shared codec space and bounded visual latent communication.

Token Communication: Token-level MAS communication is model-agnostic and inspectable but creates runtime, context-storage, and bandwidth overhead.Collaboration can also incur coordination overhead and propagate incorrect intermediate information.
Latent Communication: Training-free latent communication works best when agents share a backbone or compatible internal state formats, restricting heterogeneous teams.Learned cross-model bridges relax this assumption but add supervision and maintenance costs.
Latent Reasoning: Latent reasoning reduces visible rationale tokens for single-model deliberation, whereas heterogeneous MAS must make one model’s internal signal readable to another.The paper treats continuous traces as a communication substrate rather than only an internal reasoning mechanism.
Positioning: Vision Wormhole replaces pairwise hidden-state translation with bounded latent communication through a visually grounded codec space.Its hub-and-spoke design reduces cross-family maintenance from O(N^2) to O(N) while preserving the text-mediated role workflow.

3 Method: The Vision Wormhole

The Vision Wormhole enables heterogeneous VLM agents to communicate by converting internal reasoning into universal continuous tokens and injecting decoded messages through the receiver’s visual-token pathway. A shared reference space and label-free distillation support modular cross-agent transfer without backbone fine-tuning.

Vision Wormhole mechanism: The Vision Wormhole repurposes VLM visual inputs as a continuous communication channel for latent state transfer between heterogeneous agents.Messages bypass discrete text-token communication by entering the receiver through its vision-token span.
Vision Wormhole mechanism: Each sender extracts a latent rollout, compresses it into fixed-size universal tokens, aligns them to a reference space, and decodes them into a receiver-specific vision perturbation.The codec uses a Perceiver-style resampler, affine alignment, and a gated decoder that writes into the receiver’s image-token span.
Latent rollout: The latent rollout is a short continuous continuation of the prompt, formed by repeatedly feeding norm-calibrated pseudo-token embeddings while reusing the prompt’s attention cache.The resulting T-length sequence serves as the sender’s continuous message substrate.
Universal codec: Universal tokens have shared dimension D and bounded size, including K semantic tokens plus global and style tokens, while decoding produces Kimg receiver-specific image-query tokens.The bounded representation keeps message size independent of the sender’s reasoning length.
Codec training: Codec training uses label-free self-distillation: text communication provides the teacher, while vision injection trains codec parameters to match hidden states and next-token distributions under frozen backbones.The objective also uses RMS matching to stabilize injection magnitude in the visual embedding manifold.
Reference-space alignment: A hub-and-spoke alignment maps each model’s universal tokens to and from a reference space, requiring one map per model rather than O(N^2) pairwise adapters.Affine maps are fit with regularized least squares using a small set of shared anchor texts.

4 Experiments

The experiments evaluate Vision Wormhole against text-mediated multi-agent communication across heterogeneous VLMs, datasets, protocols, and runtime conditions. Vision Wormhole reduces runtime in most settings, produces positive macro-average accuracy changes, and shows larger gains for weaker backbones while staying closer to strong-model single-agent performance.

Experimental Settings: The evaluation spans nine reasoning benchmarks, four heterogeneous VLM families, and both two-backbone and four-backbone configurations.The benchmarks cover math and science, commonsense reasoning, and code generation; the backbones are Qwen3-VL, Gemma, SmolVLM2, and LFM2.5-VL.
Experimental Settings: TextMAS is the primary matched-prompt baseline, with identical agent roles and prompts; OCR and a heterogeneous LatentMAS-Hybrid adaptation provide additional comparisons.All configurations use a sequential Planner → Critic → Refiner → Judger workflow, while the backbones remain frozen across TextMAS and Vision Wormhole.
Main Results: Vision Wormhole reduces end-to-end runtime in most main-table cells and yields positive macro-average ∆-accuracy, with gains concentrated on MBPP-Plus and HumanEval-Plus.Table 1 reports accuracy and wall-clock time per query, with improvements expressed as ∆Acc and speedup relative to TextMAS.
Weakly Supervised Codec: With fewer than 100 anchor texts, Vision Wormhole preserves runtime gains, while accuracy changes depend on configuration.SmolVLM2/Qwen improves broadly, whereas Gemma/Qwen trades accuracy for speed on several tasks.
Single-Agent Baseline Comparison: For stronger backbones, Vision Wormhole stays closer to single-agent parity than TextMAS, while weaker backbones improve more under Vision Wormhole.The stronger-backbone comparison concerns Qwen3-VL-2B and Gemma-3-4B; the weaker-backbone comparison concerns SmolVLM2-2.2B and LFM2.5-VL-1.6B.

5 Conclusion

The paper introduces Vision Wormhole as a bounded, modular latent-communication framework for heterogeneous VLM-based collaboration. It reduces integration scaling from quadratic to linear, lowers wall-clock time in most settings, and retains positive macro-average accuracy while weak supervision preserves the runtime profile.

Conclusion: Vision Wormhole repurposes the VLM visual interface as a continuous channel for heterogeneous multi-agent collaboration.A lightweight Universal Visual Codec converts sender reasoning traces into fixed-size vision-token messages for receivers.
Conclusion: The shared reference space and hub-and-spoke design reduce multi-family integration from pairwise O(N^2) translation to O(N) scaling.Each model family uses a codec rather than maintaining pair-specific translators.
Conclusion: Vision Wormhole reduces end-to-end wall-clock time across most heterogeneous settings and improves macro-average accuracy in the controlled comparison.A weakly supervised codec trained with fewer than 100 anchor texts preserves the runtime profile, while accuracy gains remain configuration-dependent.

Limitations

The Vision Wormhole is scoped to VLM-based agents and evaluated in a research-system setting using publicly released checkpoints and specified benchmarks and protocols.

The framework applies to VLM-based agents because it uses their visual interface.
The experiments cover publicly released checkpoints, heterogeneous VLM families, and the benchmark and protocol suite described in the paper.

Ethics Statement

The work is methodological and evaluates inter-agent communication efficiency and interoperability using public benchmarks and publicly released model checkpoints.

The work focuses on improving communication efficiency and interoperability in multi-agent systems built from publicly released language and vision-language models.
Experiments use publicly available reasoning benchmarks and publicly released model checkpoints, without human subjects, private data, or personally identifiable information.

LLM Use Statement

The paper positions Vision Wormhole as a communication-interface contribution for heterogeneous MAS, complementing prior work on coordination, latent reasoning, and representation alignment.

Vision Wormhole addresses how heterogeneous agents exchange information efficiently, complementing modules that primarily address what agents do and how they act.
Prior latent-communication methods replace text with hidden states or KV-caches, but differ in whether they support homogeneous or heterogeneous model families.
Latent-reasoning research reduces token-level chain-of-thought verbosity within single models, whereas this work targets inter-model interoperability.
Tokenizer and embedding alignment methods improve transfer or inference efficiency but do not directly provide a shared latent communication substrate for MAS.
Vision Wormhole constructs a tokenizer-independent shared codec space for modular cross-family latent communication with minimal per-family adaptation.
The work replaces token exchange with bounded latent communication while retaining the same role workflow, targeting cross-family interoperability rather than a new coordination policy.

B.2 Codec Training Setup (Shared Across Runs)

The shared codec setup trains per-backbone latent-to-vision injection codecs with fixed architecture and optimization choices, while varying model, anchor coverage, and inference-time merging.

Each backbone receives a latent-to-vision injection codec, and codecs may be merged across models for communication without retraining from scratch.
The default anchor corpus contains 3,000 examples, while the weakly supervised setting contains 90 examples from the same three sources.
Training samples mini-batches uniformly from the anchor pool for 400 optimization steps, producing 800 anchor draws per model.
The codec uses D = 512 universal-token dimensions, Ku = 1024 codec tokens, Kimg = 256 injection tokens, six transformer layers, and eight attention heads.
Optimization combines hidden-state MSE, KL-style logit alignment, and an injection-statistics regularizer, with weights 1.0, 0.25, and 0.1.
Across variants, the study changes backbone, anchor size, and merge pairing while keeping codec architecture, optimizer and loss weights, step count, batch size, and sampling fixed.
Merged multi-model inference refits universal-space alignment with closed-form ridge regression instead of end-to-end retraining.
Evaluation uses greedy decoding, matched prompts and generation budgets, NVIDIA A6000 hardware, and system-level wall-clock reporting under dynamic batching.

C Additional Details: Codec Training, Alignment, and Inference

The codec extracts latent rollouts, resamples them into a universal space, aligns model-specific coordinates, and decodes them into fixed visual spans. Inference follows a bounded read–think–write cycle, while distillation matches text-channel behavior without human labels.

Codec training: NormMatch rescales hidden-state-derived pseudo-tokens to the typical token-embedding norm, reducing autoregressive instability from norm drift.The typical norm is αi = Ew∼Vi∥Ei(w)∥2, with a small ϵ added in the denominator.
Codec training: A T-step latent rollout produces Hi, which the Perceiver-style encoder compresses into a constant number of universal tokens.Learned queries attend to the variable-length rollout, yielding Ku tokens of dimension D.
Codec training: Global and style tokens communicate pooled and coarse distributional properties, stabilizing transfer across prompts and roles.The style statistic is mapped by an MLP and added to the style token.
Inference: The decoder transforms universal tokens into receiver-specific visual features and uses a learned gate to control injection strength.The gate limits over-injection for empty or low-confidence memory and adapts strength across examples.
Alignment: Affine alignment maps each model’s universal tokens into a shared coordinate system, with ridge regression fit from diverse anchor texts.Forward and reverse maps support modular composition and can be refit when models join.
Inference: The read–think–write abstraction decodes memory, runs the frozen backbone, then re-encodes intermediate rollouts for chained or independent collaboration.Message extraction costs O(T) steps and consumption costs O(L(i)img), contrasting with verbosity-dependent text communication.

E OCR-Based Image Relay Baseline

The OCR relay renders sender-generated text as an image for the receiver’s visual pathway, but it retains sender-side text generation and adds rendering and reading overhead. Across the reported settings, OCR is generally slower and less accurate than Vision Wormhole.

Baseline design: OCR renders generated text into an image that the receiver reads through its native visual input.The table reports accuracy (%) and average wall-clock time (s/query).
Results: OCR is faster than TextMAS on a macro-average basis but slower on several individual datasets and slower than Vision Wormhole overall.OCR still pays sender-side text-generation cost and adds a render-and-read step.
Results: OCR shows noticeably lower accuracy than both TextMAS and Vision Wormhole on most datasets.

F Latent Communication Failure Sweep for LatentMAS-Hybrid

The heterogeneous LatentMAS-Hybrid sweep reveals instability as latent rollouts grow, especially for cross-provider, tokenizer-heterogeneous pairs. The results motivate TextMAS as the matched-prompt baseline and distinguish Vision Wormhole’s modality-native transmission.

Setup: The sweep evaluates GSM8K accuracy and external-judge perplexity across latent-step counts for two heterogeneous cross-provider backbone pairs.It uses a fixed 200-example subset, greedy decoding, a 256-token answer budget, and batch size 4.
Failure sweep: 0.0–0.5% GSM8K accuracy and mean judge PPL from 8.1×10^5 to 8.6×10^7 characterize Qwen3-VL-2B + Gemma-3-4B across the sweep.The pair is unstable even at 16 latent steps.
Failure sweep: 14–27% accuracy and mean judge PPL around 10–12 persist for Qwen3-VL-2B + LFM2.5-VL-1.6B through 192 latent steps.At 256 steps, mean judge PPL rises to 8.1 × 10^5 and remains in the 10^6–10^7 range for longer rollouts.
Implication: The adapted latent baseline executes on heterogeneous pairs but receives high external-LM perplexity as rollouts grow, supporting TextMAS as the matched-prompt baseline.Vision Wormhole instead transmits through each recipient VLM’s modality-native image interface.

G Detailed Single vs MAS

The detailed comparison measures heterogeneous MAS against standalone single-agent baselines across datasets and configurations. Vision Wormhole generally stays closer to baseline capability than TextMAS, whose deviations often fall below baseline on stronger backbones.

Baseline construction: The single-agent baselines come from dedicated standalone runs, enabling direct comparison with combined-MAS results.
Detailed comparison: Each figure compares TextMAS and Vision Wormhole deviations from a single-agent baseline across datasets and heterogeneous MAS configurations.The baseline is represented as 0 percentage points.
Observed pattern: TextMAS frequently drops below the single-agent baseline on stronger backbones, while Vision Wormhole remains closer to parity across many settings.
Observed pattern: Vision Wormhole preserves more of the strong model’s baseline capability than text-only exchange in many cases, even when weaker models participate.The passage relates this pattern to coordination and aggregation effects in heterogeneous MAS.
Protocol: The evaluation keeps role instructions fixed across TextMAS and Vision Wormhole while varying only the inter-agent message channel.TextMAS carries text context; Vision Wormhole carries latent messages through the vision-token span.