PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning of Communication in a Mesh of Small Language Models

Mehmet Kerem Turkcan

arXiv:2609.30578v1cs.LG

TL;DR

Independent-sample majority voting saturates, while interaction can add information but risks correlated errors. TalkMesh learns communication in a decentralized mesh, and across reasoning, embodied coordination, and traffic control, messages help when another agent can provide information the acting agent cannot observe.

  • Problem

    Self-consistency saturates as majority votes over independent samples converge on the model’s most frequent answer, while interaction must add information without sacrificing vote independence.

    Source: §1 · p. 1

  • Method

    TalkMesh trains a decentralized mesh to communicate through confidence-selected hints, gated revisions, and gossip consensus.

    Source: §1 · p. 1

  • Results

    With three agents, TalkMesh matches 32-sample self-consistency; at N=32 it reaches 0.722 versus 0.492 for SmolLM3-3B on MATH-500 and 0.705 versus 0.568 for Qwen3.5-0.8B on GSM8K.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Across reasoning, embodied coordination, and traffic control, messages improve decisions when required information is unobservable to the acting agent but observable to another agent that can send it.

    Source: §1 · p. 2, §7 · p. 7

    Training requires supervision near the target distribution, traffic results use SUMO simulation under New York City timing constraints, and scaling is studied only within the Qwen3.5 family up to 4B parameters.

    Source: §6 · pp. 6–7

Abstract

from arXiv · show

Language models gain accuracy from more compute at test time, but majority voting over independent samples saturates: as samples grow, the vote converges to the model's most frequent answer. Communication can add what sampling cannot: an agent that solves a problem can pass the key step to the others. We present TalkMesh, a decentralized mesh of small language model agents that learns when and what to communicate. Each agent samples a proposal and scores it with a trained confidence head. The most confident agent broadcasts a hint; agents below a confidence threshold revise, keeping each revision that outscores its proposal. Gossip consensus approximates the vote weighted by confidence without a coordinator. A talk policy, trained with group relative policy optimization on the change in correctness after revision, writes hints and revisions. With three agents, which together generate at most six outputs, the mesh reaches the accuracy of majority voting over 32 samples with each of three models. Trained with at most 8 agents and evaluated with 32, it raises accuracy from 0.568 under self-consistency to 0.705 (Qwen3.5-0.8B, GSM8K) and from 0.492 to 0.722 (SmolLM3-3B, MATH-500). When 4 of 8 agents collude on a wrong answer with fabricated confidence and poisoned hints, majority vote accuracy falls to 0.000 (Qwen3.5-0.8B, GSM8K). A defended mesh, whose agents rescore solutions with their own confidence heads, retains 0.507. Across reasoning, embodied coordination, and traffic signal control, messages improve a decision when the acting agent cannot observe the information it requires and another agent can send it.

1 Introduction

Self-consistency saturates as votes converge on the model’s most frequent answer, while interaction can add information but risks correlated errors. TalkMesh learns when and what to communicate, and its results motivate communicating when another agent holds information the acting agent cannot observe.

1 Introduction

  • Self-consistency samples independent chains of thought and majority-votes, but accuracy saturates as the vote converges on the model’s most frequent answer.Reading peers’ reasoning can let agents correct errors and recover omitted steps, while convergent answers in debate can correlate errors and undermine majority voting’s independence.Source: §1 · p. 1
  • TalkMesh uses a confidence head to score proposals, selects the highest-scoring agent to send a hint, gates listener revisions by confidence, and uses gossip consensus to approximate a confidence-weighted vote without a coordinator.A talk policy writes hints and revisions; GRPO training rewards changes in correctness after revision.Source: §1 · p. 1
  • With three agents, trained TalkMesh matches self-consistency with 32 samples across all three models; at N=32, it reaches 0.722 versus 0.492 for SmolLM3-3B on MATH-500 and 0.705 versus 0.568 for Qwen3.5-0.8B on GSM8K.Training uses at most eight agents; communication and training each add accuracy beyond the weighted vote in the reported comparison.Source: §1 · p. 1
  • On a severe subset of ten traffic scenarios, messages reduce mean person delay by 23.6±6.7 seconds for the Qwen3.5 meshes over 60 paired episodes.The subset was fixed before language-model evaluation; it comprises ten of 70 scenarios.Source: §1 · p. 2
  • Messages improve decisions when required information is unobservable to the acting agent but observable to another agent that can send it.Withholding messages lowers performance in reasoning and traffic; in two SwarmBench settings where acting agents can observe the information, scores with and without messages agree within one standard error.Source: §1 · p. 2

2 Related work

Prior work spans inference scaling, verifier reranking, handcrafted debate, decentralized networks, and learned collaboration. TalkMesh combines learned natural-language communication with confidence-based speaker and listener selection and gossip consensus.

2 Related work

  • Self-consistency uses majority voting over sampled chains of thought, while verifier reranking and weighted voting combine verification with voting without communication.The cited prior work includes Wang et al. (2022), Cobbe et al. (2021), and Li et al. (2024).Source: §2 · p. 2
  • Unlike handcrafted discussion rounds and untrained communication policies, TalkMesh uses natural-language messages, confidence-based speaker and listener selection, listener-mean flip rewards, and in-protocol talk-policy training.The contrasted approaches include multiagent debate, ReConcile, and Exchange-of-Thought.Source: §2 · p. 2
  • TalkMesh trains its talk policy with GRPO on at most eight agents, reaches consensus by gossip without a judge model or common workspace, and exceeds self-consistency up to N=32 with five open models.The cited concurrent-work comparison notes that other teams use a common workspace or reflective prompt search.Source: §2 · pp. 2–3

3 Method

TalkMesh combines proposal scoring, a confidence-selected speaker, gated listener revisions, and gossip consensus. Its confidence head and talk policy are separate adapters on a frozen base model, and training rewards revisions according to their correctness flips.

3 Method

  • The frozen base model uses two LoRA adapters: a confidence head that scores solutions and a talk policy that writes hints and revisions.Because the base model is never updated, training leaves single-proposal accuracy unchanged.Source: §3 · p. 3
  • The five-stage protocol samples independent proposals, scores them, selects the most confident speaker by gossip, revises low-confidence listeners’ answers, and reaches a confidence-weighted answer by consensus.A revision is accepted only when its confidence score exceeds the original proposal’s; other agents retain their independent proposals.Source: §3 · p. 3
  • Gossip consensus averages sparse answer tallies with Metropolis weights, which converge to the network average on connected graphs and yield the centralized weighted vote.The weighted answer uses each agent’s confidence score.Source: §3 · p. 3
  • The confidence head is trained as an outcome verifier on sampled solutions labeled against ground-truth answers, with batches balanced between correct and incorrect solutions.The selected checkpoint has the highest validation accuracy.Source: §3 · p. 3
  • GRPO trains one talk adapter for speakers and listeners on protocol rollouts, using revision flip rewards and the mean listener flip reward for each hint.The curriculum raises rollout size K through 3, 5, and 8; problems without listeners and groups with equal rewards contribute no gradient.Source: §3 · p. 3

4 Experimental setup

The evaluation compares self-consistency, weighted voting without communication, frozen talk, and trained talk on reasoning benchmarks. It pools test-problem and proposal subsets across network sizes, with accuracy plotted against network size for three model-and-dataset combinations.

4 Experimental setup

  • The four conditions use identical proposals: self-consistency, weighted voting without communication, frozen-model talk, and trained talk.The reasoning models are Qwen3.5-0.8B on GSM8K and Qwen3-1.7B and SmolLM3-3B on numeric MATH-500 problems.Source: §4 · p. 3
  • Figure 2 plots accuracy against network size N for Qwen3.5-0.8B on GSM8K and two models on MATH-500, with one-standard-error bands and N=32 trained-talk margins over self-consistency.Except for self-consistency, conditions use the centralized weighted vote.Source: Figure 2 · p. 4
  • Reasoning proposals and revisions use temperature 0.7 with nucleus sampling and token limits of 320 for GSM8K or 448 for MATH-500.SmolLM3-3B uses nucleus 0.9; the other models use 0.8, with top-k 20.Source: §4 · p. 4
  • Pooled evaluation uses 200 test problems per model and two pools of 32 proposals per problem, testing N values from 3 to 32.Accuracy averages across random subsets and pools; standard errors combine both, and gossip consensus is compared with the centralized weighted vote and self-consistency.Source: §4 · p. 4

5 Results

TalkMesh improves collective reasoning accuracy over self-consistency, scales within the Qwen3.5 family, and retains accuracy under a coordinated attack. Across embodied coordination and traffic control, results support communication when an acting agent lacks information needed for its decision.

5.1 Collective accuracy as a function of network size

  • At N=32, trained talk raises accuracy from 0.568 to 0.705 for Qwen3.5-0.8B on GSM8K and from 0.492 to 0.722 for SmolLM3-3B on MATH-500, exceeding self-consistency at every tested N.With three agents, trained talk reaches self-consistency accuracy with 32 samples for all three models; gossip consensus agrees with the centralized weighted vote on 99.4% of problems.Source: §5.1 · p. 4, Table 1 · p. 4

5.2 Scaling within one model family

  • At N=32 on GSM8K, trained talk reaches 0.705, 0.855, and 0.865 for Qwen3.5-0.8B, 2B, and 4B, respectively, increasing with model size at every N.For Qwen3.5-2B, trained talk exceeds self-consistency by 0.362 at N=32, the largest margin reported in the paper.Source: §5.2 · p. 4, Table 1 · p. 4

5.3 Compromised agents and heterogeneous meshes

  • Under attack, the defended mesh retains 0.507, 0.340, and 0.473 accuracy for 0.8B, 1.7B, and 3B models, while majority vote falls to 0.000 accuracy for every model.On GSM8K, heterogeneous eight-agent meshes average 0.808 accuracy against 0.717 for SC, without retraining.Source: §5.3 · p. 5

5.4 Embodied coordination

  • On eighty unseen seeds, distilled Qwen3.5-0.8B matches the solver on Synchronization and raises Foraging, Pursuit, and Transport from zero to 55%, 48%, and 26% of the solver’s scores.With four agents, the Flocking EMD score rises from 0.05 to 0.60 for Qwen3.5-0.8B, from 1.65 to 2.30 for Qwen3-1.7B, and from 0.80 to 0.95 for SmolLM3-3B.Source: §5.4 · p. 5

5.5 Traffic signal control under incidents and timing constraints

  • On the severe traffic subset, delivering messages reduces person delay for every language-model mesh by a pooled 23.6 ± 6.7 s; over all scenarios, it reduces delay by 11.4 ± 6.9 s for the 4B mesh.Across all 70 scenarios, mean person delay with messages is 284.3, 282.5, and 274.6 s for the 0.8B, 2B, and 4B meshes.Source: §5.5 · p. 6, Table 3 · p. 6

5.6 An observability criterion for communication

  • Withholding messages lowers performance in reasoning and traffic, where the acting agent cannot observe required information; in two SwarmBench settings where agents can observe it, scores agree within one standard error.On Synchronization, messages do not improve the score, and directive messages in Pursuit yield 1.12 captures per episode against 1.16 without them.Source: §5.6 · p. 6

6 Limitations and future work

The evaluated tasks have automatically scored outcomes, but training requires supervision, including ground-truth answers, task rewards and a centralized solver, or controller traces. The reported scope is limited to traffic simulation under New York City timing constraints and a single model family spanning 0.8B to 4B parameters.

6 Limitations and future work

  • Training requires supervision, and the method therefore needs labelled problems near the target distribution.The paper specifies ground-truth answers in reasoning, task rewards and a centralized solver on SwarmBench, and communicating-controller traces in traffic.Source: §6 · pp. 6–7
  • Traffic results come from SUMO simulation under New York City timing constraints, and the scaling study covers only Qwen3.5 models from 0.8B to 4B.The authors identify extending the acceptance test and gossip consensus to agents with separate owners and incentives, and to networks with partitions, churn, and asynchrony, as future work.Source: §6 · p. 7

7 Conclusion

TalkMesh matches 32-sample self-consistency with three agents and improves accuracy at N=32. Its defended mesh withstands a four-agent compromise, while traffic-control results support the conclusion that messages help when another agent holds needed information.

7 Conclusion

  • 0.492 to 0.722 accuracy for SmolLM3-3B on MATH-500 and 0.568 to 0.705 for Qwen3.5-0.8B on GSM8K at N=32.With three agents, TalkMesh also reaches the accuracy of self-consistency with 32 samples.Source: §7 · p. 7
  • 0.000 majority-vote accuracy under four compromised agents versus 0.507 for the defended mesh on Qwen3.5-0.8B, GSM8K.The defended mesh significantly exceeds the undefended mesh for every model.Source: §7 · p. 7
  • 116.9 s mean person delay was lowest for the communicating traffic controller across 40 incident seeds.On the severe subset, messages reduced every Qwen3.5 mesh’s person delay by 23.6 s pooled over 60 paired episodes.Source: §7 · p. 7
  • Across settings, messages improve decisions when another agent holds information the acting agent needs.Source: §7 · p. 7

Related papers

Loading 2609.30578v1…