PaperlayerView exact revision on arXiv

Source-linked AI summary

$Σ$-Mem: An Online Reliability Memory for LLM-based Multi-Agent Systems

Peilin Feng, Suorong Yang, Soujanya Poria

arXiv:2607.27958v1cs.MAcs.AI

TL;DR

Existing multi-agent memory systems preserve interaction content but do not track which peers are reliable under particular conditions, making coordination difficult when responses are plausible or correlated. Σ-Mem maintains online competence and peer-relationship evidence, and improves peer selection, generalizes across peers and domains, and supports multiple decision mechanisms without retraining.

  • Problem

    Existing memory systems mainly preserve interaction content, while multi-agent systems need peer-specific, task-conditioned reliability evidence because central models cannot always verify plausible or correlated responses.

  • Method

    Σ-Mem maintains online historical competence and peer-relationship evidence in symmetric states updated from correctness feedback, with a shared interface for steering, routing, and weighted voting.

  • Results

    46.22% to 71.10%: Σ-Mem improves Qwen3-0.6B peer selection at CF@90, generalizes to unseen peers and domains, and improves base models in 27 of 30 cases.

  • Takeaways & Limitations

    Σ-Mem functions as a reusable reliability state for adaptive multi-agent coordination across peer sets, task distributions, and selection mechanisms without extra training.

  • Takeaways & Limitations

    Under ambiguous CF@50 evidence, Σ-Mem accumulates misleading historical signals and can steer the center model toward the wrong peer.

Abstract

from arXiv · show

Memory is central to long-horizon LLM agents, yet existing memory systems primarily preserve interaction content rather than modeling which agents can be trusted and under what conditions. This limitation is particularly important in multi-agent systems, where a central model may be unable to directly verify plausible or correlated peer responses. We introduce $Σ$-Mem, an online reliability memory that records historical competence evidence for individual peers and peer relationship evidence across the peer set. Both forms of evidence are maintained as real symmetric states and updated from post-decision correctness feedback. By Weyl's inequality, the spectral change caused by each event-level update is bounded, enabling stable online adaptation without retraining the underlying models. $Σ$-Mem provides a general write-and-read interface: the same memory can be used for residual steering of a central model, response-free peer routing, or reliability-weighted voting. Across five Qwen-family models, $Σ$-Mem adapts to counterfactual reliability shifts and generalizes to unseen peers and task domains. Direct memory readouts also outperform majority voting and the best fixed peer over the full OOD evaluation set. Moreover, performance improves consistently as more correctness feedback becomes available, indicating that $Σ$-Mem progressively accumulates actionable reliability information. These results establish reliability memory as a reusable foundation for adaptive coordination in LLM-based multi-agent systems.

1 Introduction

The introduction identifies a gap in content-focused agent memory: multi-agent systems also need persistent, task-conditioned evidence about which peers are trustworthy and when. Σ-Mem addresses this gap with online competence and relationship states that support stable, reusable coordination across changing peers, domains, and decision mechanisms.

  • Motivation: Existing memory systems preserve historical interaction content, but multi-agent systems also need reliability memory to track who can be trusted and when.Peer agreement can reflect shared biases or correlated errors rather than independent evidence, while central models may be unable to verify responses.
  • Method: Σ-Mem records two complementary forms of evidence: each peer’s historical competence under task conditions and relationships among peers.Both evidence types are updated online from external correctness feedback.
  • Results: The same frozen memory supports steered peer selection, response-free peer routing, and reliability-weighted voting without extra training.These mechanisms enable reuse of the recorded reliability state for different coordination decisions.
  • Method: Σ-Mem uses real symmetric states with decayed, bounded updates, so Weyl’s inequality limits each event’s eigenvalue change while consistent evidence accumulates.This design prevents abrupt distortions and attenuates inconsistent or isolated signals over long horizons.
  • Results: At CF@90, Σ-Mem raises Qwen3-0.6B from 46.22% to 71.10% when the reliable peer changes.Across five Qwen-family central models, it also improves with four- and five-peer pools containing unseen peers and beyond training domains.

2 Background and Preliminaries

The section establishes real symmetric matrices and Weyl’s inequality as the spectral basis for stable Σ-Mem updates. It also explains residual steering as a way to influence attention through hidden states while keeping attention parameters frozen.

  • Spectral preliminaries: A real symmetric matrix satisfies M = M^T and has the structure required for applying Weyl’s eigenvalue-stability analysis.Weyl’s inequality concerns perturbations of real symmetric matrices.
  • Spectral preliminaries: A single event-level symmetric update has bounded spectral norm, limiting how much any memory eigenvalue can change.Σ-Mem uses this controlled-perturbation property to motivate stable online spectral evolution.
  • Residual steering: In Transformer sequence modeling, self-attention computes query, key, and value matrices from the residual stream.The hidden sequence is denoted X.
  • Residual steering: Residual steering adds a peer-specific steer δ_p(x) to the residual stream, changing the hidden states used by attention without modifying frozen attention weights.The steer depends on the task input sequence x.

3 Σ-Mem Mechanism

Σ-Mem maintains complementary competence and peer-relationship evidence as symmetric online states, updating them from external correctness feedback. These states support peer-conditioned central-model evaluation as well as response-free routing and reliability-weighted voting.

  • Memory states: Σ-Mem records individual peer competence in symmetric matrices {M_p} and peer relationships in a symmetric graph matrix G.M_p captures historical task-conditioned competence, while G captures relationships across peers’ correctness signals.
  • Online updates: Each evaluated event updates competence using a normalized direction, decay factor γ, and rank-one symmetric evidence term scaled by η.Persistent aligned evidence accumulates spectrally, whereas isolated or noisy evidence is bounded by Weyl’s inequality and damped by decay.
  • Online updates: Theorem 1 shows persistent competence eventually dominates event-level noise in M_p’s leading eigenvalue with high probability.The noise term does not grow with T, while the signal term saturates at ημ_p/(1−γ), so sufficiently long histories reflect genuine competence.
  • Steered peer selection: At decision time, peer-specific memory readouts are projected into the center model and injected into upper decoder blocks, producing distinct residual steering for each peer.The resulting attention computations are peer-conditioned, allowing candidate evaluation to use both current answers and historical reliability records.
  • Response-free decisions: Direct memory readouts support response-free routing and M-weighted voting, with scores computed from the current question and historical competence evidence before answer aggregation.Routing selects the highest-scoring peer, while voting aggregates reliability weights for peers sharing a canonical answer; response content is not semantically judged.

4 Experiments

Experiments show that Σ-Mem improves adaptive peer selection across counterfactual reliability shifts, unseen peers, and out-of-distribution tasks. Its reliability evidence also transfers across routing and voting mechanisms and benefits from broader correctness feedback.

  • Experimental setup: Σ-Mem is trained on 2,963 task events spanning mathematical reasoning, retrieval-augmented question answering, and code generation, then evaluated across four experimental axes.Each event contains a task input, peer-generated answers, and external per-peer correctness labels.
  • Counterfactual reliability shifts: At CF@90, Σ w/ G raises Qwen3-0.6B accuracy from 46.22% to 71.10%, countering the base model’s fixed preference for peer 1.The base model’s accuracy falls from 61.01% at CF@0 to 46.22% at CF@90 as the counterfactual attack intensifies.
  • Counterfactual reliability shifts: Σ w/ G generally outperforms Σ w/o G, showing that peer relationship evidence adds complementary information beyond individual historical competence.The relationship state models how peers’ correctness patterns relate to one another.
  • Peer-count generalization: Σ-Mem generalizes to enlarged peer pools with unseen peers, improving accuracy over the base model at CF@0, CF@70, and CF@90.The mechanism adds new memory states without modifying shared learned parameters; CF@50 remains an exception because misleading historical evidence steers selection incorrectly.
  • OOD generalization: Σ-Mem improves over the corresponding Qwen base model in 27/30 OOD cases, including BBH gains from 20.38% to 28.66% for Qwen3-4B and from 19.29% to 28.17% for Qwen3-8B.The experiments cover unseen commonsense, knowledge, science, complex-reasoning, and language-understanding benchmarks.
  • Selection mechanisms: Direct historical-evidence routing and weighted voting outperform majority voting and best-fixed-peer baselines across the full OOD dataset, demonstrating mechanism-agnostic reliability memory.M-Vote performs better for Qwen3, while Σ w/G achieves the best overall accuracy with stronger Qwen3.5 models.

5 Related Work

Related work spans computational trust and reputation in multi-agent systems, LLM routing and ensembling, and persistent memory for LLM agents. These lines address trust signals, model selection or combination, and long-term state persistence.

  • Computational Trust and Reputation in MAS: Computational-trust methods maintain per-agent trust from interaction outcomes and incorporate multiple, social, or contextual reputation sources for coordination.Examples include TRAVOS, FIRE, and REGRET.
  • LLM Routing and Ensembling: LLM routing and ensembling methods select or combine models using predicted quality, cost, reward signals, candidate rankings, or confidence-weighted voting.Examples include RouteLLM, FrugalGPT, Zooter, LLM-Blender, and ReConcile.
  • Memory for LLM Agents: Persistent-memory systems for LLM agents manage virtual context hierarchies, retain long-term records across sessions, and control what state re-enters context during long-horizon execution.Examples include MemGPT, MemoryBank, Mem0, DeltaMem, and proactive memory agents.

6 Conclusion · A Theoretical Analysis · A.1 Proof of Weyl’s Inequality

Σ-Mem records individual and relational peer competence in symmetric states, enabling stable online updates and reuse across multiple coordination mechanisms without retraining. The theoretical appendix proves the eigenvalue perturbation bound underlying this stability analysis.

  • 6 Conclusion: Σ-Mem records historical competence evidence for individual peers and peer relationships across the peer set.Its state design is symmetric.
  • 6 Conclusion: The same reliability memory supports residual steering, peer routing, and weighted voting through a general write-and-read interface.These uses do not require retraining the underlying models.
  • A.1 Proof of Weyl’s Inequality: The proof considers real symmetric matrices M and E in R^r×r and orders the eigenvalues of each matrix.This establishes the setting for the eigenvalue perturbation bound used in Eq. 1.
  • A.1 Proof of Weyl’s Inequality: The i-th eigenvalue is characterized using the Courant–Fischer variational principle.The argument applies this variational characterization to compare the eigenvalues of the perturbed and unperturbed matrices.
  • A.1 Proof of Weyl’s Inequality: For every unit vector v, the spectral norm definition supplies the bound needed in the eigenvalue comparison.This norm bound controls the effect of the perturbation E.
  • A.1 Proof of Weyl’s Inequality: The reverse inequality follows by writing M = (M + E) + (−E) and applying Eq. 23 again.This supplies the complementary direction of the perturbation bound.
  • A.1 Proof of Weyl’s Inequality: Combining the two inequalities yields the eigenvalue perturbation bound for every i = 1, . . . , r.The resulting inequality is |λ_i(M + E) − λ_i(M)| ≤ ∥E∥_2, which proves Eq. (1).

A.2 Proof of Theorem 1 · B Dataset Construction

Theorem 1 is proved by combining bounded per-event spectral perturbations, a signal-noise decomposition, and persistent-alignment analysis. The result shows that accumulated competence signal can dominate a horizon-independent noise floor, while larger memory decay improves stationary signal-to-noise under constant competence but slows adaptation to drift.

  • A.2 Proof of Theorem 1: Each event changes any eigenvalue of the decayed memory by at most η.The update is rank one with spectral norm η, so Weyl’s inequality bounds the eigenvalue shift.
  • A.2 Proof of Theorem 1: The competence model separates task-dependent expected competence from independent, zero-mean event-level noise bounded in [−2, 2].Correctness remains c_p,t ∈ {−1, 1} because the noise is defined as ξ_t = c_p,t − μ_p(x_t).
  • A.2 Proof of Theorem 1: Projected memory decomposes into a deterministic competence signal and a stochastic noise contribution along any fixed reference direction.The decomposition follows by unrolling the memory recursion and substituting c_p,t = μ_p(x_t) + ξ_t.
  • A.2 Proof of Theorem 1: The noise contribution remains bounded by O(η/√(1 − γ^2)) with high probability, independently of the horizon T.Thus, reliability depends on whether the deterministic signal grows beyond this fixed noise floor.
  • A.2 Proof of Theorem 1: Under persistent alignment and constant competence bias, the signal is a geometric series that converges to ημ_p/(1 − γ).The stationary signal-to-noise ratio increases with γ and diverges as γ → 1−.
  • A.2 Proof of Theorem 1: Increasing γ improves stationary signal-to-noise under constant competence but slows forgetting and adaptation when peer competence drifts.This is a one-sided memory-length/adaptivity trade-off under the time-invariant competence assumption.
  • A.2 Proof of Theorem 1: Once the signal exceeds the noise floor, the leading eigenvalue is bounded away from zero according to the peer’s true competence rather than event-level noise.The spectral conclusion follows by applying the variational characterization of eigenvalues to the fixed-direction projection.

B.1 Training Dataset

Σ-Mem is trained on a fixed offline dataset of 2,963 task events spanning mathematics, retrieval-augmented question answering, and coding. Each task receives responses from three fixed peer models, whose binary correctness labels drive event-level memory updates.

  • Dataset composition: 2,963 task events comprise 1,000 GSM8K, 1,000 SQuAD, and 963 APPS training examples across three competence domains.The domains are mathematics, retrieval-augmented question answering, and coding.
  • Peer responses: Three fixed peers—Gemma-3-4B-it, Phi-4-mini-instruct, and Qwen2.5-Coder-7B-Instruct—independently generate one response per task.The peer responses are evaluated for correctness on the training dataset.
  • Memory updates: Binary correctness labels from each peer response are used for event-level memory updates.Table 5 summarizes each peer’s training-dataset accuracy.

B.2 Counterfactual Attack Dataset

The counterfactual attack dataset spans nine benchmarks across mathematics, retrieval-augmented question answering, and coding. It systematically swaps strong-peer and weak-peer responses while preserving peer identities and evaluates Σ-Mem with up to five peers, including unseen models.

  • Attack construction: An event qualifies for attack construction when the strong peer is correct and at least one remaining peer is incorrect.The strongest model is identified from clean accuracy for each constituent dataset.
  • Attack construction: The attack swaps the strong peer’s complete response and correctness label with an incorrect weak peer while preserving peer identities and metadata.The attack ratio r controls the counterfactual attack condition described in the construction protocol.
  • Peer-set generalization: Σ-Mem is evaluated with four and five peers by adding Llama-3.2-3B-Instruct and BitCPM-CANN-3B without modifying learned parameters.Both expanded settings retain the same 2,685 task events, domain composition, and counterfactual attack ratios as the original evaluation stream.

B.3 Generalization Beyond Training Domains

Σ-Mem generalizes beyond its mathematics, RAG, and coding training domains by mapping unseen tasks into a learned competence-direction space. Evaluation covers five OOD benchmarks spanning commonsense, knowledge, science, complex reasoning, and natural-language understanding.

  • Generalization mechanism: Σ-Mem reuses historical reliability evidence from related training domains for unseen tasks through 𝝓(x).The memory is trained only on mathematics, RAG, and coding events, yet maps new tasks into the learned competence-direction space.
  • OOD evaluation: The OOD evaluation tests physical commonsense reasoning with PIQA and broad-domain knowledge understanding with MMLU.These benchmarks are explicitly outside the training domains.
  • OOD evaluation: The evaluation also covers science with SciQ, complex reasoning with BBH, and natural-language understanding with SuperGLUE.SuperGLUE accuracy is computed by pooling its six evaluated subsets by instance count.
  • Results: Table 9 summarizes peer accuracy across the unseen domains and identifies the best-performing peer on each benchmark.The reported benchmarks extend beyond the mathematics, RAG, and coding domains used for training.

C Direct Memory Readout under Counterfactual Streams · D SuperGLUE breakdown

Direct memory readout routes peers from question-derived reliability evidence without observing current peer responses, outperforming response-based routing in many weak-judge counterfactual settings, while stronger judges reverse that pattern. Across SuperGLUE subtasks, Σ-Mem generally improves aggregation, especially where Qwen3 centers struggle to identify the correct peer.

  • C Direct Memory Readout under Counterfactual Streams: M-route selects the peer with the highest question-conditioned reliability score from peer-specific memory matrices, without using current peer responses.Correctness labels are used only afterward to update memory.
  • C Direct Memory Readout under Counterfactual Streams: For Qwen3 centers, M-route matches or exceeds Σ w/G in 9 of 12 settings, with margins up to 5.81 points and 6.56 points at CF@90.The reported margins are for Qwen3-0.6B and Qwen3-8B, respectively.
  • C Direct Memory Readout under Counterfactual Streams: For Qwen3.5 backbones, Σ w/G surpasses M-route in 6 of 8 settings, and the remaining M-route advantage shrinks.At CF@90, M-route leads by 1.26 points for Qwen3.5-4B.
  • C Direct Memory Readout under Counterfactual Streams: Bayesian Beta B1 reaches 51.43%, 58.18%, 67.26%, and 75.23% from CF@0 to CF@90, while Σ w/G exceeds it by 12.44–21.34 points at CF@0.At CF@50, Σ w/G also exceeds B1 for four of five center models.
  • C Direct Memory Readout under Counterfactual Streams: At CF@70 and CF@90, B1 selects Qwen-Coder 2,459 and 2,626 times, becoming competitive as the counterfactual shift dominates.The Qwen3.5 variants with Σ w/G remain above B1 at both settings.
  • C Direct Memory Readout under Counterfactual Streams: Memory-only routing is preferable with weak judges or stable reliability patterns, whereas response-based judgment becomes essential when history is ambiguous and the central model is stronger.The Bayesian comparison indicates Σ-Mem’s gains are not reducible to global Bayesian reputation tracking.
  • D SuperGLUE breakdown: Σ-Mem improves aggregation accuracy on most SuperGLUE subtasks, with particularly large gains for Qwen3 models on SuperGLUE-RTE and SuperGLUE-CB.These gains address cases where base centers fail to identify the correct peer, using historical competence and peer-relationship evidence.
  • D SuperGLUE breakdown: A few SuperGLUE cases degrade slightly when the base model is already highly accurate, because historical steering can alter an otherwise correct judgment.Figure 4 decomposes each subtask into base accuracy and the change introduced by Σ-Mem, with oracle accuracy as an upper bound.

E Validation of Historical Competence in M

This section validates direct readout of historical competence evidence on the same OOD datasets and fixed peers across five Qwen-family central models. The experiments isolate whether task-conditioned competence evidence provides reliable signals, with per-benchmark results reported for each model.

  • Experimental setup: Direct-readout experiments evaluate historical competence evidence using the decision rules defined in Sec. 3.4.The section reports per-benchmark results for the direct-readout experiments.
  • Experimental setup: The diagnostic uses the same OOD datasets and three fixed peers described in Sec. B.3.These comparisons are designed to isolate the contribution of task-conditioned historical competence evidence.
  • Model coverage: Results cover Qwen3-0.6B, Qwen3-4B, Qwen3-8B, Qwen3.5-4B, and Qwen3.5-9B as central models.Separate tables report the best-performing result on each benchmark for every model.
Loading 2607.27958v1…