Let Models Speak Ciphers: Multiagent Debate through Embeddings

1 INTRODUCTION

• Debate gains have often depended on state-of-the-art models, while smaller open-source models have not shown consistent benefits.The introduction cites examples involving self-repair and bargaining feedback that worked for powerful models but not weaker ones.Source: §1 · p. 1
• Natural-language debate is interpretable, but sampling one token can discard information in the model’s belief over the vocabulary.The most confident token can be wrong, as illustrated by generating “9” instead of “6”.Source: §1 · pp. 1–2
• CIPHER bypasses token sampling by sending weighted averages of vocabulary embeddings, preserving richer information and enabling nearest-neighbor interpretation.The paper says this richer signal is particularly useful when a model is uncertain.Source: §1 · p. 2
• In the debate setup, agents answer independently, then refine their answers using peers’ responses before reaching a final answer in most cases.Source: §1 · p. 2
• 0.5 −5.0% performance improvement over the natural-language counterpart is reported across five reasoning datasets, with smaller models also outperforming majority voting.The datasets span GSM8K, Arithmetic, MMLU Formal Logic, MMLU High School Math, and MMLU Professional Psychology.Source: §1 · p. 2
• CIPHER targets LLMs sharing a tokenizer, including models with different embedding-vocabulary mappings, and the paper includes an extensive mechanism-focused ablation study.Source: §1 · pp. 2–3

2 RELATED WORK

• Du et al. (2023) frame multiagent debate as peer-driven iterative refinement, while efficacy on smaller open-source models remained underexplored.Source: §2 · p. 3
• Madaan et al. (2023)’s self-improvement framework uses an LLM as both generator and critic, but Vicuna-13B struggled to produce consistently formatted feedback.Source: §2 · p. 3
• Code self-debugging uses natural-language explanations and unit-test results; weaker-model self-repair did not beat majority voting (Olausson et al., 2023).Source: §2 · p. 3
• Wei et al. (2022)’s CoT was extended into Tree-of-Thought (Yao et al., 2023; Long, 2023) and Graph-of-Thought (Besta et al., 2023); CIPHER uses CoT and few-shot CoT.Source: §2 · p. 3

3.2 COMMUNICATION THROUGH SEMANTIC EMBEDDINGS

• CIPHER semantic embeddings approximate, but do not precisely match, token embeddings, and generation stops at a nearest-neighbor EOS embedding or the maximum sequence length.The EOS condition or the length limit ends a response generation process after τ steps.Source: §3 · p. 4, §3.2 · p. 5

3.1 NATURAL LANGUAGE COMMUNICATION

• Natural-language generation samples one token from a vocabulary distribution at each step, compressing the model’s belief and discarding information about other tokens.The distribution is treated as the model’s belief about the most appropriate next token.Source: §3.1 · p. 4

3.2 COMMUNICATION THROUGH SEMANTIC EMBEDDINGS

• CIPHER replaces sampled next tokens with probability-weighted vocabulary embeddings, retaining richer output-belief information for direct inter-model communication.The weighted average is used in place of sampled tokens because models may not grasp vectors outside the tokenizer embedding space’s convex hull.Source: §3.2 · p. 4, §3.1 · p. 4
• In debate, agents receive the prompt and prior CIPHER responses as concatenated embeddings, refine responses each round, then convert and aggregate final outputs.Conversion uses nearest-neighbor search over the vocabulary; the prompt includes the instruction, question, and possible responses from earlier rounds.Source: §3 · p. 3, §3.2 · p. 5
• Temperature controls probability-distribution smoothness; Bayesian optimization selects each method’s best temperatures for evaluation, followed by sensitivity analysis.As temperature approaches zero, both methods become greedy; at large temperatures CIPHER averages broadly while natural-language generation samples broadly.Source: §3.2 · p. 5
• CIPHER’s weighted averaging parallels Expected SARSA’s expectation over possible actions, while natural-language token sampling parallels vanilla SARSA.The paper frames autoregressive generation as a Markov decision process with the vocabulary as the action space.Source: §3.2 · p. 5

4.1 EXPERIMENTAL SETUP

• The baselines are Single Answer, Self-Consistency (Majority@5), and Natural Language Debate (NLD), the closest comparison to CIPHER.The authors omit self-improvement via feedback because open-source models have difficulty generating appropriately formatted critic feedback.Source: §4.1 · p. 6
• Experiments mainly use LLaMA2-70B and evaluate five reasoning datasets spanning grade-school math, arithmetic, formal logic, high-school math, and professional psychology.Additional models include LLaMA-65B, Falcon-40B-Instruct, MPT-30B, and WizardMath-70B-V1.0; selected large datasets use separate 200-question validation and test samples.Source: §4.1 · p. 6
• Following Du et al. (2023), the evaluation uses three debate rounds between two LLMs, with few-shot chain-of-thought and zero-shot step-by-step prompting.Accuracy is measured on final answers for debates among identical models; for different models sharing a tokenizer, correctness is assessed for all final-round answers.Source: §4.2 · p. 6, §4.1 · p. 6

4.2 RESULTS

• CIPHER outperforms NLD by 1.0–5.0% across five reasoning datasets in debates between identical LLaMA-family models.The comparison uses three-round debates; Table 1 reports accuracies, with five responses per question for each baseline except Single Answer.Source: §4.2 · p. 6, Table 1 · p. 7
• LLaMA-65B’s Arithmetic accuracy rises from 35% to 62.5% with CIPHER in debates with LLaMA2-70B.Both agents benefit from debating, and CIPHER yields higher accuracies for both than NLD.Source: §4.2 · p. 7, Table 2 · p. 7
• On GSM8K, CIPHER adds 0.5%–3.5% performance across Falcon-40B-Instruct, MPT-30B, and WizardMath-70B-V1.0.The experiments use open-source models without modifying their weights.Source: §4.2 · p. 7, Figure 3 · p. 7

5 ANALYSIS AND DISCUSSION

• The analysis covers extended debate scales, temperature sensitivity, and partial CIPHER generation.The paper also places additional positional-bias and performance-bound experiments in Appendix B.Source: §5 · pp. 7–8

5.1 DEBATE IN EXTENDED SCALES

• Adding debate rounds yields small gains at the cost of more responses, while adding debaters is also beneficial with a similar trade-off.The figure reports round results on Arithmetic and debater-count results on GSM8K, both with LLaMA2-70B.Source: §5.1 · p. 8, Figure 4 · p. 8

5.2 TEMPERATURE SENSITIVITY

• CIPHER performs best with diverse temperatures, pairing a low-temperature agent with a high-temperature agent and using the lower-temperature agent’s answer.For CIPHER, the higher-temperature agent contributes less-confident token information, while lower-temperature agents are needed for human-interpretable results.Source: §5.2 · p. 8, Figure 5 · p. 9

5.3 ABLATION STUDY ON CIPHER

• Partial CIPHER using entropy or maximum-probability uncertainty closely tracks full CIPHER, while reversing either trigger causes large performance drops.CIPHER is applied when uncertainty exceeds a threshold and greedy sampling is used otherwise; the reverse variants apply CIPHER at lower uncertainty.Source: §5.3 · pp. 8–9

• CIPHER enhances debate efficacy across reasoning tasks without special training, outperforming majority voting and natural-language debate methods.The paper describes results across a wide range of LLMs.Source: §6 · p. 9
• The study highlights LLM belief information as a potential source of value for debate and a first step toward unlocking that potential.The authors describe this work as closing the gap between proprietary and open-source models.Source: §6 · p. 9
• More efficient ways to transfer belief information remain an open question because LLMs are trained to intake embeddings of natural-language tokens.Source: §6 · p. 9

LIMITATIONS AND BROADER IMPACTS

• CIPHER currently applies only to LLMs sharing a common vocabulary set; using distinct tokenizers would require challenging alignment of their embedding-vocabulary mappings.Tokenizers can segment the same word differently, such as into subwords or characters versus a single token.Source: LIMITATIONS AND BROADER IMPACTS · p. 10

A MULTIAGENT DEBATE FRAMEWORK

• In natural-language debate, agents begin with independent text answers, iteratively receive prompts concatenated with peer responses, and aggregate final-round answers directly.Unlike CIPHER, this process does not require a nearest-neighbor search over the vocabulary to aggregate the final responses.Source: A MULTIAGENT DEBATE FRAMEWORK · p. 13

B.1 INVESTIGATING POSITIONAL BIAS IN DEBATES

• Swapping response order has negligible effects for similarly tempered debaters but can cause significant differences when their temperatures diverge.Both natural-language debate and CIPHER perform better with more diverse debaters.Source: Figure 7 · p. 13, §B.1 · p. 13

B.2 PERFORMANCE BOUNDS OF DEBATES

• Debate can be detrimental for LLaMA2-7B on GSM8K, but poses little harm for the more powerful LLaMA2-70B.The experiments compare random, misaligned, single-answer, CIPHER, and expert debaters to explore lower and upper performance bounds.Source: Figure 8 · p. 14, §B.2 · p. 14

B.3 ATTENTION HEATMAPS

• CIPHER’s heatmap has bright spots around the 40th attention head and 74th time step, unlike NLD’s uniform distribution.The authors suggest the concentrated areas may be more relevant to the task.Source: §B.3 · p. 14, Figure 9 · p. 14
• The heatmaps measure similarity between the first agent’s last-token query and keys from its preceding 100 tokens at LLaMA2-70B’s 45th decoder layer.Source: §B.3 · p. 14
• The comparison uses the Arithmetic question from Figure 1 during the final debate round.Source: §B.3 · p. 14

C QUALITATIVE RESULTS

• Figure 10 shows a complete NLD debate on Arithmetic between two LLaMA2-70B debaters at temperatures (0.00, 0.67).Source: Figure 10 · p. 15, C QUALITATIVE RESULTS · p. 15
• Figure 11 shows a complete CIPHER debate on Arithmetic between two LLaMA2-70B debaters at temperatures (0.25, 1.75), with embeddings converted to language by nearest-neighbor search.Source: Figure 11 · p. 15, C QUALITATIVE RESULTS · p. 15
• Figure 12 presents a CIPHER debate on GSM8K using two LLaMA2-70B debaters at temperatures (0.40, 1.00).Source: C QUALITATIVE RESULTS · p. 15, Figure 12 · p. 16

D.1 TEMPERATURE

• Temperature settings for LLaMA-family debates correspond to the evaluations in Table 1 and are listed in Table 3.Source: §D.1 · p. 16, Table 3 · p. 16
• Temperature settings for LLaMA2-70B and LLaMA-65B on Arithmetic and GSM8K correspond to Table 2 and appear in Table 4.Source: §D.1 · p. 16, Table 4 · p. 17
• Table 5 lists temperatures for cross-model two-agent GSM8K debates; Tables 6 and 7 list LLaMA2-70B temperatures for extended-scale GSM8K debates.Source: §D.1 · p. 17, Table 5 · p. 17, Table 6 · p. 17

D.2 COMPUTATION RESOURCE

• LLaMA-family debates use four NVIDIA A100 SXM 80GB GPUs as the major computation resource.Source: §D.2 · p. 17

E.1.1 INITIAL ROUND PROMPT

• Initial math prompts request step-by-step reasoning and a numerical final answer; most models receive three-shot examples, while WizardMath-70B-V1.0 uses an author-provided chain-of-thought prompt.The examples include arithmetic questions and a question placeholder.Source: §E.1.1 · p. 18

E.1.2 DEBATE ROUND PROMPT

• Debate-round math prompts ask models to use another agent’s solution as additional information and update their own answer; WizardMath-70B-V1.0 uses a zero-shot prompt.Examples show agents’ solutions, the model’s solution, and a correct solution before a response is generated.Source: §E.1.2 · pp. 19–20

E.2.1 INITIAL ROUND PROMPT

• Initial multiple-choice prompts ask for step-by-step reasoning and a letter answer, with worked examples involving algebra, fractions, and exponential growth.The prompt specifies the final-answer format as “Answer: {letter}.”Source: §E.2.1 · pp. 20–21

E.2.2 DEBATE ROUND PROMPT

• Multiple-choice debate prompts ask models to use another agent’s solution to update their answer; examples include fixed-point and function-value questions.The supplied examples include both conflicting agent responses and a worked correct solution.Source: §E.2.2 · pp. 22–23

E.3.1 INITIAL ROUND PROMPT

• Initial psychology prompts use worked, step-by-step multiple-choice answers, with examples on criterion-referenced measurement, author-gender bias, therapy ethics, and the keyword method.The instruction asks for a letter as the final answer.Source: §E.3.1 · pp. 23–24

E.3.2 DEBATE ROUND PROMPT

• Psychology debate prompts ask models to revise using another agent’s response; examples include author-gender attribution and crowding questions.The gender-attribution example shows an agent answer, a competing answer, and a correct solution.Source: §E.3.2 · pp. 24–25
• A crowding debate example explains why age is the exception among factors affecting responses to crowding and gives B as the answer.The example contrasts age with self-esteem, gender, and culture.Source: §E.3.2 · pp. 25–26

E.4.1 INITIAL ROUND PROMPT

• Initial logic prompts ask for step-by-step reasoning and a final answer as a letter.Examples include symbolizing an English sentence and evaluating consistency with a truth table.Source: §E.4.1 · pp. 26–27
• The initial logic examples also ask readers to choose a consistent valuation or interpret a quantified proposition in English.The sample solutions select B for consistency and D for the English interpretation.Source: §E.4.1 · pp. 26–27

E.4.2 DEBATE ROUND PROMPT

• The debate prompt directs an agent to use another agent’s solution as additional information before revising its answer.In the symbolization example, the agents give different choices, while the supplied correct solution identifies A.Source: §E.4.2 · pp. 27–28
• A second logic debate example contrasts agents’ proposed valuations with a supplied solution that concludes the proposition set is inconsistent.The supplied solution says no truth-table row makes all propositions true.Source: §E.4.2 · pp. 28–29

E.5.1 INITIAL ROUND PROMPT

• The initial arithmetic prompt requests step-by-step calculation and a numerical final answer.Its examples show worked expressions ending in 380, 168, and -160.Source: §E.5.1 · p. 29

E.5.2 DEBATE ROUND PROMPT

• The prompt asks an agent to update its answer using another agent’s solution as additional information; one example contrasts answers of 300 and 380, then gives 380 as the result.A separate expression is evaluated as 11.Source: §E.5.2 · pp. 29–30