Source-linked AI summary
Learning What to Retain: Gated-Memory Routing for Efficient Collaboration in Multi-Agent LLM Systems
Rakibul Hasan Rajib, Mengxing Zheng, Qian Lou
TL;DR
Multi-agent routing must adapt to evolving execution states without forcing each decision to process redundant history. Gated-Memory Routing uses learned write and retrieval gates, role and backbone routing, and adaptive halting over selective execution memory; across five benchmarks, it achieves the best average accuracy while reducing HumanEval inference cost relative to the strongest baseline.
Problem
Query-only routing cannot adapt to intermediate progress or errors, while full-history routing processes redundant or low-utility prior steps and inflates cost.
Method
Gated-Memory Routing jointly learns memory writing and retrieval with role, backbone, and halting decisions over a compact, task-relevant execution state.
Results
Across MATH, GSM-Hard, MBPP, HumanEval, and MMLU-Pro, the framework achieves the best average accuracy, exceeding the strongest baseline by 2.44 points and reducing HumanEval inference cost by 31.9%.
Takeaways & Limitations
Selective execution memory lets multi-agent routing condition decisions on filtered task-relevant state while adapting collaboration depth and cost.
Takeaways & Limitations
Evaluation uses closed-domain tasks with verifiable answers, and efficiency measurements beyond parameter count cover only two benchmarks for FLOPs and wall-clock analysis.
Abstract
from arXiv · showhide
Large language model (LLM)-based multi-agent systems tackle complex reasoning by orchestrating how multiple agents are configured and how they collaborate. A central challenge is to adapt orchestration to the evolving collaboration state. Routing from the query alone cannot adapt to intermediate progress or errors, which hurts accuracy. Routing from the complete execution history supplies this missing context, but forces later decisions to process every prior step, including redundant or low-utility ones. This creates an execution-history overload that inflates cost. Effective orchestration instead requires a compact state that captures useful progress without accumulating redundant context. We propose Gated-Memory Routing, which conditions each decision on the query and a learned execution memory. A learned Memory Write Gate commits only non-redundant reasoning steps, and a learned Retrieval Gate supplies each agent a compact, relevant subset, so every decision conditions on a clean, informative state. At each step, the system selects the next role and backbone from this memory, while an Adaptive Halting Controller stops execution once the memory contains sufficient evidence for answering. Across five reasoning and code-generation benchmarks, our framework is both effective and efficient: it attains the best average accuracy, exceeding the strongest baseline by 2.44 points, while reducing HumanEval inference cost by 31.9% relative to that baseline. Code is available at https://github.com/rajibrhasan/gated-memory-routing
1 Introduction
The paper frames multi-agent orchestration as a routing problem: query-only routing lacks execution feedback, whereas full-history routing overloads decisions with growing redundant context. Gated-Memory Routing addresses this trade-off with selective memory, adaptive routing, and halting, achieving stronger accuracy with lower cost.
- Motivation: Multi-agent performance depends on orchestration choices including agent count, roles, backbones, and accessible context.The paper treats these choices as the routing problem governing whether collaboration produces reasoning gains or redundant computation.
- Motivation: Query-only routers cannot respond to partial progress, errors, or emerging solution gaps because they commit decisions before execution.Execution-history routing adds responsiveness but retains accumulated trajectory content with limited filtering.
- Motivation: Full-history routing increases context length and cost by forcing later decisions to process an unfiltered, growing trajectory.The paper identifies redundant, noisy, or low-utility intermediate content as the source of execution-history overload.
- Approach: Gated-Memory Routing conditions decisions on a learned selective execution memory rather than query-only input or raw history.Its memory uses write and retrieval gates to preserve task-relevant reasoning and expose filtered state to downstream routing.
- Approach: The framework jointly routes roles and backbones while gating memory writes and retrieval and adaptively halting when further collaboration is unlikely to add value.This couples memory curation, context selection, collaboration structure, and reasoning depth within one routing framework.
- Results: 2.44 average points and 31.9% lower HumanEval inference cost distinguish the method from the strongest baseline across five benchmarks.The evaluated tasks are MATH, GSM-Hard, MBPP, HumanEval, and MMLU-Pro.
3 Methodology
Gated-Memory Routing frames multi-agent orchestration as sequential routing over a learned, selective execution memory. Its gates control role, backbone, retrieval, writing, and halting so decisions use relevant progress while execution depth adapts to the query.
- Framework formulation: The sequential process maintains a gated subset of execution records and selects a role, backbone, retrieved context, and write decision at each step.Execution terminates when halting fires or the maximum depth is reached, after which an aggregator produces the final answer from the terminal memory.
- Framework formulation: Routing conditions on the query and selective memory rather than the query alone or the full raw trajectory.This state lets role allocation respond to the retained solution state instead of every intermediate step.
- Framework components: The framework combines history-aware role allocation, role-conditioned backbone routing, retrieval gating, memory writing, and adaptive halting in one end-to-end trajectory.The allocator and router choose agents and models from the current state, while retrieval and writing regulate the information they use and preserve.
- Memory operations: The Retrieval Gate selects a step-adaptive subset of stored records instead of exposing all memory or using a fixed top-k context.Independent record decisions allow context size to contract toward empty or expand when several records are relevant.
- Memory operations: The Memory Write Gate admits a new record only when it is relevant and novel relative to stored records, using a learned stochastic relevance–redundancy criterion.Its learned coefficient balances relevance against redundancy, and the first record is admitted on relevance alone when memory is empty.
- Adaptive computation: The Adaptive Halting Controller makes reasoning depth a consequence of execution by deciding after each step whether memory contains sufficient evidence for aggregation.It carries a recurrent state summarizing post-write memory across steps, separately from the query-attended routing context.
4 Experimental Setup
The experiments evaluate the framework across mathematical reasoning, program synthesis, and knowledge-intensive question answering using heterogeneous open-weight backbones and diverse orchestration baselines.
- Datasets and benchmarks: Evaluation spans five benchmarks: GSM-Hard, MATH, HumanEval, MBPP, and MMLU-Pro.The tasks cover mathematical reasoning, program synthesis, and knowledge-intensive question answering.
- Baselines: Baselines include single-model inference, single-agent reasoning, fixed-topology collaboration, adaptive workflow optimization, evolving orchestration, and query-only routing.This set compares against both static and adaptive multi-agent systems.
- LLM backbones: The router selects among five open-weight LLM backbones ranging from 3B to 32B parameters.Backbone prices are proportional to parameter count as a hardware-independent proxy for per-token inference compute.
- Training configuration: Training uses Adam with learning rate 0.01, GRPO group size G = 6, and 16 queries per batch.The default maximum depth is ϕ = 6, with cost, entropy, and VAE coefficients specified for optimization.
- Evaluation controls: Fixed-topology multi-agent baselines use six agents, matching the framework’s maximum depth budget ϕ = 6.The experiments also use the same 26 heterogeneous role profiles as MASRouter.
5 Results
Gated-Memory Routing delivers the strongest overall accuracy across five benchmarks while reducing inference cost through selective memory and adaptive halting. Its gains persist against query-only and full-history routing, with performance-cost trade-offs depending on task structure and reasoning depth.
- Overall performance: Gated-Memory Routing achieves the best average accuracy and leads on four of five benchmarks, improving over the strongest baseline by 2.44 points.MBPP is the exception, where fixed multi-agent configurations perform best.
- Accuracy–cost trade-off: 43.86% lower HumanEval cost than MASRouter and 31.9% lower cost than Puppeteer accompany higher accuracy than both baselines.The method lies on the high-accuracy end of the HumanEval accuracy-cost Pareto frontier.
- Efficiency mechanism: Selective memory and adaptive halting reduce redundant context and computation while preserving answer quality.Halting stops easy queries early, whereas gated memory supplies compact context to downstream agents.
- Ablation study: Removing the LLM Router causes the largest accuracy loss, including a 14.0-point drop on GSM-Hard.Removing the role allocator also causes a smaller but consistent accuracy decrease.
- Ablation study: Disabling both memory gates lowers accuracy by 2.56 points on GSM-Hard and 6.25 points on HumanEval, while disabling halting raises cost by over 40% and more than doubles it, respectively.Without halting, training rollout steps increase by about 1.8× on both benchmarks.
- Depth sensitivity: At maximum depth ϕ = 6 on MATH, most accuracy gains are realized; increasing depth to ϕ = 10 adds under one point while cost continues rising.The default depth is near the knee of the accuracy–cost curve.
6 Conclusion
Gated-Memory Routing coordinates multi-agent systems through learned gated execution memory rather than query-only or raw-history routing. Across five benchmarks, it combines higher average accuracy with lower HumanEval inference cost.
- Gated-Memory Routing conditions routing decisions on learned gated execution memory instead of the query alone or full raw history.Memory Write and Retrieval Gates keep the state compact and task-relevant for role and backbone routing.
- An Adaptive Halting Controller uses the gated memory to adapt reasoning depth and cost to each query.
- 2.44 points: the method exceeds the strongest baseline in average accuracy across five benchmarks.
- 31.9%: HumanEval inference cost is reduced relative to the strongest baseline.
Limitations
The implementation uses frozen encoders and backbones while jointly training routing, gating, halting, and embedding parameters. The method's state and retrieval mechanisms encode retained reasoning and select compact memory records for routing.
- A frozen all-MiniLM-L6-v2 encoder maps queries, descriptions, and agent responses into 384-dimensional vectors.Response embeddings are detached before entering memory, and encoder parameters remain fixed during training.
- Role and backbone descriptions are mapped by small VAEs into 128-dimensional latent embeddings used by the routing modules.Their reconstruction and KL terms contribute the LVAE regularizer.
- A two-layer Transformer encodes the query with retained reasoning tokens into the routing context vector.Each token combines role and backbone latents with response-content modulation, and query attention produces a pooled history vector.
- The Retrieval Gate admits records using learned scaled cosine-similarity logits, with memory capped at ϕ records.The bias controls typical memory length, while Bernoulli sampling determines record admission.
- The router is trained with a group-relative score-function policy-gradient estimator while the backbones remain frozen.A single optimizer updates routing, gating, halting, and embedding parameters jointly.
9 Cost Calculation for Open-Weight Backbones
The framework assigns open-weight backbones size-based reference prices and uses them to estimate trajectory cost. Its algorithm routes roles and models, writes memory selectively, and aggregates the final answer.
- Reference input and output rates are set to 0.003N and 0.010N per million tokens, with N denoting parameter count in billions.Output is weighted more heavily to reflect autoregressive decoding cost.
- The algorithm takes a query, role set, model pool, and maximum depth ϕ, then encodes the query before routing.
- At each step, the system performs role and model routing and samples a Memory Write Gate decision from the write score.
- The final answer is produced by an aggregator using the most frequently selected backbone across the execution.
- Trajectory cost is the additive sum of per-step backbone costs based on input and output token usage.The proxy abstracts away attention cost, batching, and memory bandwidth.
- Approximate FLOPs and batch-amortized wall-clock time are reported for HumanEval and MBPP as hardware-relevant corroboration.The FLOPs estimate uses 2N per processed token and omits the sequence-length-dependent attention term.
10 Per-Dataset Accuracy and Cost
The per-dataset analysis reports size-based reference costs for GSM-Hard and HumanEval and compares accuracy–cost frontiers. Gated-Memory Routing reaches the high-accuracy end of the GSM-Hard frontier at lower cost than key baselines.
- GSM-Hard and HumanEval tables report per-method accuracy and total test-set cost using the size-based reference cost.
- The GSM-Hard accuracy–cost frontier is presented as the companion to the HumanEval frontier.
- Highest accuracy: the method reaches the top accuracy among compared GSM-Hard methods at substantially lower cost than the strongest multi-agent and routing baselines.Inference cost is shown on a log scale.
11 Variance across Training Seeds
Across three-seed retraining, the method remains strongest on all five benchmarks, while adaptive halting reduces realized computation during training.
- Variance across Training Seeds: The method has the highest mean accuracy on all five benchmarks across three independent training seeds.Table 8 reports per-benchmark means and sample standard deviations, plus the macro-average across benchmarks.
- Variance across Training Seeds: 4.91 points is the method’s overall-mean improvement over MASRouter across the three-seed comparison.
- Variance across Training Seeds: Write rates remain strictly between 0 and 1, while retrieval becomes more selective as memory grows.These patterns indicate that the trained gates do not collapse to always-write, never-write, always-retrieve, or never-retrieve policies.
- Variance across Training Seeds: Per-step gate-distribution entropy stays well above zero throughout training on all three benchmarks.
- Variance across Training Seeds: Adaptive halting lowers training-time computation by reducing trajectory lengths and rollout steps per update.Table 10 compares mean realized depth and total rollout steps with and without halting.
14 Agent Role Profiles
The system uses heterogeneous role profiles spanning coding, commonsense or knowledge, and mathematical reasoning, while performance remains similar with a reduced role set.
- Agent Role Profiles: The role allocator selects among 26 heterogeneous profiles grouped into 7 code, 7 commonsense/knowledge, and 12 mathematical-reasoning roles.Each profile is encoded from a natural-language description into the latent role space.
- Agent Role Profiles: 70.74 on GSM-Hard and 89.06 on HumanEval were achieved with 13 subsampled roles, versus 70.55 and 89.84 with all 26 roles.The authors report that performance does not hinge on retaining the full role set.
- Agent Role Profiles: The catalog includes specialized roles such as TestAnalyst, AlgorithmDesigner, ProgrammingExpert, BugFixer, ReflectProgrammer, PlanSolver, and ProjectManager.These profiles respectively emphasize testing feedback, algorithm design, implementation, correction, reflection, pseudocode, and code-structure oversight.
- Agent Role Profiles: Commonsense and knowledge profiles include KnowledgeExpert, Reflector, Critic, Scientist, Economist, Historian, and WikiSearcher.Their descriptions cover question answering, re-examination, critique, scientific explanation, economics, historical reasoning, and entity lookup.
- Agent Role Profiles: Mathematical and coding profiles include MathSolver, Mathematician, MathTeacher, MathAnalyst, Inspector, Engineer, and CertifiedAccountant.The profiles span hint-based solving, arithmetic and planning, teaching, symbolic derivation, verification, engineering reasoning, and financial calculations.
15 Aggregator Prompt
After halting, an Aggregator LLM combines the query with retained execution records and produces the final task-specific answer.
- Aggregator Prompt: After the halting controller stops execution, the Aggregator LLM weighs retained agent analyses and results, identifies errors, and commits to one answer.Its backbone is the model selected most frequently by the router during the trajectory.
- Aggregator Prompt: For GSM-Hard, the aggregator must return a single pure number and end with “The answer is <number>”.The prompt disallows units, symbols, commas, scientific notation, and unnecessary rounding.
- Aggregator Prompt: For HumanEval and MBPP, the aggregator returns the full implementation in one Python code block.The prompt asks it to use the function signature, docstring, and any prior designs or implementations.
- Aggregator Prompt: For MMLU-Pro, the aggregator selects one option from up to ten choices and ends with “The answer is X”.
16 Backbone LLM Profiles
The LLM router chooses among five open-weight backbones using encoded descriptions that represent both capabilities and reference input/output prices.
- Backbone LLM Profiles: The LLM router selects among five open-weight backbones through natural-language capability descriptions mapped into a latent backbone space.
- Backbone LLM Profiles: llama-3.2-3B is the cheapest and fastest option, suited to easier queries where quality is less critical.It is described as a compact 3-billion-parameter instruction-tuned model with solid general reasoning and instruction following.
- Backbone LLM Profiles: llama-3.1-8B is a low-cost, general-purpose reasoning and instruction-following baseline across a broad range of tasks.
- Backbone LLM Profiles: mistral-nemo-12B is a mid-size workhorse offering multilingual performance, a 128k-token context window, and solid reasoning and coding at modest cost.
- Backbone LLM Profiles: qwen-2.5-14B targets moderately hard queries with strong mathematics, coding, and general reasoning for its size class.
- Backbone LLM Profiles: qwen-2.5-32B is reserved for the hardest queries, offering frontier-level performance among open-weight models in its size class.