Source-linked AI summary
Dual Latent Memory for Visual Multi-agent System
Xinlei Yu, Chengming Xu, Zhangquan Chen, Bo Yin, Cheng Yang, Yongbo He, Yihao Hu, Jiangning Zhang, Cheng Tan, Xiaobin Hu, Shuicheng Yan
TL;DR
Existing VMAS can lose accuracy as agent turns increase, while token use rises. L2-VMAS replaces text exchange with dynamically synthesized, separate latent memories for perception and thinking, retrieved through entropy-driven on-demand triggering. Across four benchmarks, it improves average accuracy over VMAS in both Instruct and Thinking models.
Problem
In VMAS, increasing agent turns does not reliably improve accuracy: performance peaks at turn 3 and later declines as token use rises.
Source: §1 · p. 1Method
L2-VMAS uses shared, dynamically synthesized latent memories that decouple perception and thinking, with entropy-driven triggering to retrieve them when needed.
Source: §1 · p. 2Results
Across four benchmarks, L2-VMAS improves average accuracy over VMAS by 1.3-3.7% in Instruct models and 2.2-5.4% in Thinking models.
Source: §4.2 · p. 7Takeaways & Limitations
Abstract
from arXiv · showhide
While Visual Multi-Agent Systems (VMAS) promise to enhance comprehensive abilities through inter-agent collaboration, empirical evidence reveals a counter-intuitive "scaling wall": increasing agent turns often degrades performance while exponentially inflating token costs. We attribute this failure to the information bottleneck inherent in text-centric communication, where converting perceptual and thinking trajectories into discrete natural language inevitably induces semantic loss. To this end, we propose \textbf{L}$\mathbf{^{2}}$\textbf{-VMAS}, a novel model-agnostic framework that enables inter-agent collaboration with dual latent memories. Furthermore, we decouple the perception and thinking while dynamically synthesizing dual latent memories. Additionally, we introduce an entropy-driven proactive triggering that replaces passive information transmission with efficient, on-demand memory access. Extensive experiments among backbones, sizes, and multi-agent structures demonstrate that our method effectively breaks the "scaling wall" with superb scalability, improving average accuracy by 2.7-5.4% while reducing token usage by 21.3-44.8%.
1. Introduction
The introduction argues that text-centric VMAS hit a scaling wall: deeper collaboration can degrade accuracy while increasing token costs. L2-VMAS addresses this with shared dual latent memories and entropy-driven, on-demand retrieval.
1. Introduction
- Text-centric communication loses fine-grained visual details and hidden cognitive states, while conflating perception and thinking can cause interference that worsens in deeper turns.The introduction identifies accumulated semantic loss and duplicate tokens as contributors to performance degradation and computation growth.Source: §1 · pp. 1–2
- Existing VMAS accuracy declines after an early peak and falls below the single-agent baseline by turn 6, while total token consumption exceeds 30 times the baseline by turn 10.The authors attribute these failures to information loss and interference in text-centric transmission.Source: §1 · p. 1
- L2-VMAS replaces natural-language exchange with shared dual latent memories that separately represent perception and thinking and are dynamically synthesized across previous turns.The framework is described as model-agnostic.Source: §1 · p. 2
- Entropy-driven triggering retrieves the two memories only when necessary, shifting collaboration from passive reception to on-demand orchestration.The introduction also reports validation across five VLM backbones, four model sizes, and six multi-agent structures, with higher average accuracy and lower token consumption.Source: §1 · p. 2
2. Requisite Analyses
The analyses document a scaling wall in existing VMAS: deeper turns raise token use while accuracy eventually declines, and full-content transmission fares poorly in later turns. The authors summarize the resulting concerns as performance degradation and the limitations of text-centric communication that conflates perception and thinking.
2. Requisite Analyses
- Experiments use MMBench with Qwen3-VL-8B-Thinking; accuracy and token use are tracked across turns, with turn 1 as the single-agent baseline.The authors describe accuracy as task effectiveness and token use as integrated computational and communicative efficiency.Source: §2 · p. 2, §2.1 · p. 2
2.1. Analytical Experiments
- Accuracy peaks at the 3rd turn, rising from 84.8 to 86.6 (2.1% relative improvement), then falls below single-agent performance by turn 6; token use reaches 16,840 at turn 10.At turn 10, accuracy is 2.6% below the single-agent baseline; token use is 557 at turn 1 and 5,241 at turn 5.Source: §2.1 · pp. 2–3, Figure 2 · p. 3
- Conclusion-only transmission gains 0.5% to 1.1% across successive turns, whereas full-content transmission gains 0.7% at turn 2 but loses 3.8% by turn 10.At later turns, full-content transmission underperforms both perception-only and thinking-only configurations.Source: §2.1 · p. 3, Figure 3 · p. 3
2.2. Findings
- The findings characterize existing VMAS as suffering performance degradation and exponential token overhead in deep turns, while text-centric interaction that mixes perception and thinking is an inferior transmission medium.The authors connect these issues to compromised reliability and scalability.Source: §2.2 · p. 3
3. Methodologies
L2-VMAS adds a model-agnostic external memory system to VMAS, with separate perception and thinking memories synthesized across agent turns. Entropy-driven orchestration retrieves relevant memory on demand, alongside a staged training scheme for the external components.
3.1. Preliminary
- The model-agnostic external dual-memory system attaches to a directed VMAS and separates memory synthesis from on-demand orchestration.The base agents remain frozen; the design targets efficiency, proactivity, and scalability through decoupling, entropy-triggered access, and dynamic synthesis.Source: §3.1 · pp. 3–4
3.2. Memory Synthesis
- Perception memory combines fine-to-coarse visual observations, while thinking memory stores high-entropy-delimited semantic chunks in a capacity-limited bank that prunes or merges units.Three granularity levels—local, regional, and global—balance performance and computational overhead; thinking-memory management uses triggering rate and semantic similarity.Source: §3.2 · pp. 4–5
3.3. Memory Orchestration
- During decoding, an adaptive entropy window triggers memory access; a learned gate selects perception or thinking memory, which is retrieved, refined, and injected at the trigger position.The trigger rule bars two continuous triggers within a window of length W.Source: §3.3 · pp. 5–6
3.4. Training Recipe
- Training keeps each base VLM frozen and uses three RL-driven stages: random memory activation, isolated orchestration training, then joint training of external memory components.The stages target memory construction and updates, recall, and end-to-end performance, respectively.Source: §3.4 · p. 6
4. Experiments
Experiments compare L2-VMAS with single-agent and conventional text-communication baselines across models, benchmarks, sizes, and agent structures. Results report accuracy, token use, memory effectiveness, scalability, and generalization.
4.1. Settings
- The evaluation covers five VLM backbones, eight visual benchmarks, four model sizes, and six multi-agent structures, with memory abilities trained on GQA without test-benchmark exposure.Experiments use dynamic structure for the reported main results; implementation details also specify three-stage training and the memory-triggering settings.Source: §4.1 · p. 6
4.2. Main Results
- Across five backbones, L2-VMAS improves average accuracy by 2.7-5.4% and reduces total token consumption by 21.3-44.8%; augmented visual scenarios show over 4% average performance enhancement.Table 1 compares five base models on accuracy and token usage, with average results in the rightmost column.Source: §4.2 · pp. 6–7, Table 1 · p. 7
- Across 2B/4B/8B/32B models, gains over VMAS are 1.3-3.7% for Instruct and 2.2-5.4% for Thinking; across six topologies, gains average 2.7-6.3%.L2-VMAS consistently outperforms the corresponding baseline across the tested topologies.Source: §4.2 · p. 7, Table 2 · p. 7, Table 3 · p. 8
4.3. Additional Analyses
- Perception memory improves perception-task accuracy by 5.5% versus 1.4%, thinking memory improves thinking-task accuracy by 8.2% versus 1.8%, and combined memories gain 7.9% on mixed tasks.The memory triggering curves are reported as self-adaptive across tasks.Source: §4.3 · p. 8, Figure 5 · p. 8
- As agent turns increase, L2-VMAS gains 13.9% for Instruct and 19.2% for Thinking over VMAS; at turn 10 it exceeds turn 1 by 13.6% and 15.1%, respectively.The reported VMAS baselines decline after three turns and fall below the single-agent level by turn 10.Source: §4.3 · p. 8, Figure 6 · p. 8
- Ablations examine memory synthesis and orchestration, sensitivity analyses assess hyperparameter effects on efficiency and effectiveness, and unseen-benchmark tests report greater generalization than baselines.The analyses are reported in Tables 6-10.Source: §4.3 · p. 8
5. Conclusion
The conclusion identifies a scaling wall in conventional VMAS and presents L2-VMAS as a dual latent-memory approach with entropy-driven triggering. The related-work discussion distinguishes VMAS research from latent-visual methods that do not address VMAS-specific challenges.
5. Conclusion
- Conventional VMAS exhibit a scaling wall: deeper agent turns can worsen performance while increasing computation cost; L2-VMAS decouples perception and thinking in dual latent memory and adds entropy-driven triggering and retrieval.The authors describe the framework as a scalable path toward more reliable and complicated VMAS.Source: §5 · pp. 8–9
A.1. Visual Multi-agent System
- VMAS research spans task-adaptive visual-agent frameworks, training, planning, theory, benchmarks, and applications, while most existing paradigms retain text-centric communication (Wang et al., 2023; Elhenawy et al., 2024; Huang et al., 2025a).The passage also lists work on structured visual understanding, medical analysis, and embodied intelligence.Source: §A.1 · p. 14
A.2. Visual Latent Space
- Latent visual reasoning, inserted visual tokens, and reusable visual-memory methods are reviewed, but the passage says none directly applies to VMAS because they do not address its efficiency, proactivity, and scalability challenges.Examples include LaCoT, Mirage, and CoMEM (Sun et al., 2025; Yang et al., 2025b; Wu et al., 2025a).Source: §A.2 · p. 14
B.1. Experimental Details
The experimental details define accuracy and token-use measurements across agent turns and describe structured extraction of perception, thinking, and conclusion from agent outputs.
B.1. Experimental Details
- Task accuracy is the percentage of correctly answered tasks, while total token usage sums tokens used and generated by each agent per turn, including prompt, visual-feature, instruction, and output tokens.Agent turns range from 1 to 10; one turn is the independent-model, single-agent baseline.Source: §B.1 · p. 14
- Inter-agent transmission conditions include conclusion-only, perception plus conclusion, thinking plus conclusion, and full-content communication; GPT-5.1 is used to extract content structurally.Full-content communication includes unstructured perception, thinking trajectories, and the conclusion.Source: §B.1 · p. 14
- The extraction prompt separates directly observable visual facts from reasoning or planning, and separates both from the final task answer, returning text spans with start and end positions.The output also includes ambiguity status and a reason.Source: §B.1 · p. 15
B.2. Additional Results
Additional experiments across four visual benchmarks report consistent conclusions about accuracy, total token usage, and inter-agent transmission comparisons.
B.2. Additional Results
- MMbench, MMStar, RealWorldQA, and SimpleVQA are used to check whether the earlier findings hold across data distributions.The experiments compare accuracy and token use among agents and inter-agent information transmission.Source: §B.2 · p. 15
- Figure 7 compares accuracy and total token usage across agent turns on four benchmarks.The passage identifies the comparison scope but does not provide figure values.Source: Figure 7 · p. 15, §B.2 · p. 15
- Figure 8 compares inter-agent transmission content on four benchmarks.The passage reports the same overall conclusion across these benchmarks without specifying the figure's individual outcomes.Source: §B.2 · p. 15, Figure 8 · p. 16
C. Methodologies
The method compresses previous-turn information into separate perception and thinking memories, then uses entropy and a learnable gate to route memory access. Its compressors use masked attention, while visual features are aligned through the VLM’s native projector.
C.1. Compression Module
- Compression queries attend to content tokens, and their final embeddings become the compressed outputs; content tokens cannot attend back to the queries.The three compressors share a lightweight transformer structure, using masked self-attention followed by residual, layer-normalization, and feed-forward updates.Source: §C.1 · p. 16
C.2. Visual Encoding and Alignment
- Visual features from multiple granularity levels are projected into language space and concatenated into the perception-memory value.The method reuses the VLM’s native projector, kept frozen during training, rather than adding alignment parameters.Source: §C.2 · p. 17
C.3. Entropy
- Entropy over the language model’s word-generation probabilities is used to identify semantic boundaries and decide when to trigger memory access.The probability distribution is defined over vocabulary words at each time step.Source: §C.3 · p. 17
C.4. Learnable Gate
- A learnable gate uses a hidden-state window to select perception or thinking memory, with Gumbel-Sigmoid training producing near-discrete routing decisions.Temperature annealing makes routing smoother early in training and sharper later; inference can threshold the relaxed gate for a deterministic choice.Source: §C.4 · pp. 17–18
C.5. Training Recipe
The VLM backbones remain frozen while training updates the external dual-memory modules through a three-stage PPO recipe. The stages separately train memory synthesis, memory access, and then their coordinated adaptation.
C.5. Training Recipe
- The pretrained VLM backbones stay frozen while external dual-memory modules are trained using a three-stage PPO recipe with accuracy as the primary reward.The stated aim is to preserve backbone generalization while directing learning toward memory synthesis and orchestration.Source: §C.5 · p. 18
- Stage I trains memory synthesis and updating, using random memory triggers while unlocking the compressor and merging compressor.Random triggering encourages frequent writes and updates during memory-encoding training.Source: §C.5 · p. 18
- Stage II freezes synthesis and trains on-demand access; Stage III unlocks all memory components for coordinated adaptation.Stage II unlocks the gate, reused compressor, and refining compressor, while Stage III enables end-to-end adaptation.Source: §C.5 · p. 18
D.1. Settings
The experiments compare single-agent decoding and text-communicating VMAS baselines across six multi-agent structures. The VMAS settings largely follow prior multi-agent configurations.
D.1. Settings
- The single-agent baseline decodes independently, while VMAS agents pass generated outputs sequentially as input instructions to later agents.The final agent’s output is the system response to the original task.Source: §D.1 · p. 18
- Experiments use linear, layered, centralized, random, complete, and task-adaptive dynamic multi-agent structures.Dynamic structures use variational graph autoencoders to generate task-adaptive topologies.Source: §D.1 · pp. 18–19
- The training recipe and configurations are listed in Table 4, while Table 5 reports Qwen3-VL-8B results on multi-image and video benchmarks.The base VMAS environments largely follow prior multi-agent settings.Source: Table 4 · p. 19, Table 5 · p. 19, §D.1 · p. 19
D.2. Additional Results
Across multi-image and video benchmarks, L2-VMAS improves on the Single and VMAS baselines; it also reports gains at higher agent turns and on unseen datasets. The results include adaptive memory triggering and some small accuracy drops in near-ceiling settings.
D.2. Additional Results
- Small accuracy drops can occur on relatively straightforward benchmarks or smaller models already near a performance ceiling, while token reductions remain significant.The authors describe these drops as isolated and low-magnitude; aggregate results still show consistent average gains and token reductions across backbones and model scales.Source: §D.2 · p. 19
- 4.0% (Instruct) and 4.6% (Thinking) are the average gains across MuirBench, BLINK, MVBench, and LVBench versus Single and VMAS baselines.The reported improvements span both multi-image and video-based tasks.Source: §D.2 · pp. 19–20
- On unseen datasets, L2-VMAS improves average accuracy by 6.4%; the full-component model performs best on all benchmarks in the ablation.Removing Attribution and Thinking produces the most dramatic accuracy declines; sensitivity analyses report an accuracy–efficiency trade-off for the tested configurations.Source: §D.2 · p. 20, Table 6 · p. 20, Table 10 · p. 21
- Perception memory prevails on perception-focused tasks, thinking memory on reasoning tasks, and the two fluctuate in a balanced interplay on mixed tasks.The reported triggering patterns indicate memory allocation varies with task demands.Source: §D.2 · p. 20, Figure 10 · p. 22
- At the 10th turn, performance gains are 6.0–13.9% for the Instruct model and 9.2–19.2% for the Thinking model across four benchmarks.The comparison is between L2-VMAS and VMAS scalability.Source: §D.2 · p. 20, Figure 11 · p. 22