Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View

• The test bed uses GPT-3.5 on MATH, MMLU, and Chess Move Validity, with four societies of three agents varying easy-going and overconfident traits.Source: §1 · pp. 1–2
• Across multi-round strategies, agents use debate or reflection, with either all agents sharing a thinking pattern or one agent using a different pattern each round.Source: §1 · p. 2
• Substantive debate improves collaboration, while heterogeneous agent traits show no clear performance difference; uniform thinking patterns improve efficiency, and adding agents or rounds does not consistently improve outcomes.Source: §1 · p. 2
• LLM agents exhibit conformity and majority-rule behavior in group thinking, echoing social psychology theories.Source: §1 · p. 2
• The authors use social-psychology theories to contextualize LLM collaboration and argue rational small-group strategies may be more efficacious than relying on scale alone.Source: §1 · p. 2

2.1 Preliminary Concepts in Collaboration

• Agents vary by easy-going or overconfident traits and use debate or reflection, with multi-round strategies defined as permutations of those thinking patterns.Debate involves exchanging ideas and reaching consensus; reflection reviews prior responses and refines answers. These patterns can unfold over several rounds.Source: §2.1 · pp. 3–4

2.2 Society Simulation

• The primary study constructs four three-agent societies ranging from totally overconfident to totally easy-going, with eight possible three-round strategies.All agents employ the same thinking pattern within each collaboration round; later analysis considers more agents, rounds, and strategies.Source: §2.2 · p. 4

2.3 Experimental Settings

• Evaluation covers high-school multiple-choice questions (50 randomly selected), math problems (50 randomly chosen from Levels 3–5), and chess-move validity, predicting a legitimate subsequent move for a specified piece.The datasets are MMLU (Hendrycks et al., 2021a), MATH (Hendrycks et al., 2021b), and BIG-Bench (Srivastava et al., 2022); ChatGPT serves as the LLM agent. Results report five-trial average accuracy and standard deviations, WIN-TIE frequency for matching or exceeding the continuous-debate baseline (Du et al., 2023b), and average token costs.Source: §2.3 · pp. 4–5

(1) Societies do not clearly differ in performance but differ significantly in their tendency

• Society traits make little difference to accuracy, but society composition affects consensus, with all-easy-going societies more likely to agree.The authors attribute the indistinct accuracy differences to LLM alignment inhibiting extreme overconfidence.Source: (1) Societies do not clearly differ in… · p. 5
• Debate-initial strategies generally outperform other strategies across datasets, while debate-dominant strategies are also superior overall.The order of thinking patterns significantly affects effectiveness.Source: (1) Societies do not clearly differ in… · p. 6
• Collaborative-strategy effectiveness depends on task and difficulty: continuous debate often performs well, while debate plus continuous reflection suits difficult MATH level 5.Strategy differences are slight for Chess Move Validity, unlike MMLU and MATH.Source: (1) Societies do not clearly differ in… · p. 6

3.2 Impact of Machine Society Settings

• Odd-sized societies generally outperform even-sized ones, and three agents offer the best balance of performance and efficiency.More agents are more likely to reach consensus, while accuracy differences among odd agent counts are indistinctive.Source: §3.2 · p. 6, Figure 3 · p. 7
• Three-round collaboration is relatively effective and efficient, although the effect of round count depends on strategy, especially for reflection-starting or reflection-dominant strategies.Performance fluctuates across 3 to 10 rounds, but the variations are not especially remarkable overall.Source: §3.2 · p. 6, Figure 4 · p. 7
• Assigning different thinking patterns to agents within the same round tends to hurt performance, so consistent patterns across agents maximize collaborative efficacy.The study randomly allocates patterns while excluding rounds in which all agents use the same pattern.Source: §3.2 · p. 7, Figure 5 · p. 7

4 Phenomena of Conformity and Consensus Reaching

• Conformity can turn an initially correct agent toward a wrong answer, while debate can bring all agents to the right answer.The first case resembles groupthink’s detriments; the second reflects benefits in groupthink and SoM, as varied viewpoints resolve conflicts toward informed consensus.Source: §4 · pp. 7–8
• Conformity is widespread and generally rises across rounds, benefiting ChatGPT and Qwen 72B but harming LlaMA2 Chat 13B/70B and Mixtral 8×7B.Source: §4 · p. 8
• As rounds increase, conformity’s benefits weaken while its detriments strengthen.Source: §4 · p. 8, Figure 6 · p. 8
• Reflection generally increases consensus-cluster counts, making consensus harder, whereas debate is more likely to reach consensus.Source: §4 · p. 8, Figure 7 · p. 8

5 Conclusion and Future Work

• LLM agents show impressive collaboration capabilities across different individual traits, thinking patterns, and collaborative strategies.Source: §5 · p. 8
• Agents’ human-like behaviors resonate with social psychology theories and highlight the potential of human-AI interaction.Source: §5 · p. 8
• Future work should refine collaboration behavior, explore multi-agent societies more deeply, and apply social psychology insights to socially aware NLP systems.Source: §5 · p. 8

Limitations

• The study does not examine societies combining different LLMs because of expense, although differently distributed pretraining data and strategies aligned with human intentions may yield social-level findings.The authors suggest that such differences could lead to more interesting findings.Source: Limitations · p. 9
• The study searches possible scenarios but does not let agents autonomously adapt their collaboration strategies to specific scenarios.The authors also note that debate can approach the upper limit but consumes more resources, while other strategies may perform better with less overhead.Source: Limitations · p. 9
• The experimental setup is relatively straightforward, omitting a broader range of traits and larger-scale societies.Source: Limitations · p. 9
• Performance is evaluated through manual validation and rule-based matching, limiting assessment of more realistic and creative tasks such as literary creation.Source: Limitations · p. 9

Ethics Statement

• The authors report that the study used publicly accessible datasets, contained no proprietary or confidential information, and raised no ethical concerns.They state that the research followed high ethical standards and best practices.Source: Ethics Statement · p. 9

Appendices A–D

• Appendices A–D cover key takeaways, related work, potential real-world applications, and implementation details, including setup, evaluation, and collaboration illustrations.Implementation details are in §D, with experimental setup in §D.1, evaluation in §D.2, and an illustration of agent collaboration in §D.3.Source: Overview of Appendices · p. 16

Appendices E–G

• Appendices E–G examine machine social collaboration, society settings, and social-psychological themes of conformity, consensus reaching, and group dynamics using ChatGPT.§G separates conformity and consensus reaching (§G.1) from group dynamics (§G.2).Source: Overview of Appendices · p. 16

Appendices H–I

• Appendices H–I analyze different backbone LLMs and assess prompt effectiveness; backbone analyses cover LlaMA2 Chat 13B and 70B, Qwen 72B, and Mixtral 8×7B.The model-specific analyses appear in §§H.1–H.4, respectively.Source: Overview of Appendices · p. 16

A Key Takeaways

• Debate-led collaboration performs relatively well and total reflection poorly; difficult tasks favor debate plus continuous reflection, while simple tasks can use self-consistency or reflection.The takeaways describe debate as optimal when it starts or dominates collaboration, and totally-reflection p1p1p1 as generally worst.Source: A Key Takeaways · p. 16
• Overconfident agents lose that trait in groups; three agents, three collaboration rounds, and uniform within-round thinking patterns are identified as effective settings.Three agents are described as advantageous for performance and cost, and three rounds as effective and efficient.Source: A Key Takeaways · p. 16
• Collaboration is generally effective, especially on difficult tasks, but it can induce beneficial or harmful conformity whose benefits decline and detriments increase with more rounds.The takeaways also report these collaboration and conformity patterns across multiple figures and models.Source: A Key Takeaways · p. 16
• Totally easy-going societies are more likely to reach consensus; debate helps consensus reaching, while reflection impedes it.Source: A Key Takeaways · p. 16

B Related Work

• Prior LLM-agent research emphasized agent components and explored debate or reflection, while AgentVerse patterns inform this study’s experiments with additional societies and strategies (Du et al., 2023b).The related work also notes that this framework can extend to agents with different model compositions, although that method is not demonstrated here (Chen et al., 2023c).Source: B Related Work · pp. 16–17
• Related work describes socially aligned LLM agents simulating human behaviors, roles, and collaboration, and connects multi-agent collaboration with social-psychology phenomena including conformity and consensus (Duéñez-Guzmán et al., 2023).Source: B Related Work · p. 17

C Potential Real-world Applications

• Potential applications include using LLM agents for social research, negotiation and mediation, AI ethics and governance, and team collaboration tools.These applications draw on agents’ ability to simulate social interactions, offer fair solutions, inform responsible AI guidelines, and facilitate teamwork.Source: C Potential Real-world Applications · p. 17
• Other proposed applications include tutoring, healthcare coordination, crisis management, content creation, and interactive entertainment.Source: C Potential Real-world Applications · p. 17

D.1 Experimental Setup

• LLaMA2 Chat 13B and 70B experiments cap collaboration at 4 rounds and 4 agents, while experiments varying agent number for other LLMs use Chess Move Validity.Some cases exceeding the 4096-token context limit are handled by pruning content from earlier rounds; the other-model restriction reflects the cost of five replicates across eight strategies.Source: §D.1 · p. 18
• The collaboration-round analysis examines 8 representative strategies because the number of viable strategies grows exponentially with each added round.The passage gives 10 rounds yielding 210 unique strategies as an example.Source: §D.1 · p. 18
• For gpt-3.5-turbo-1106, temperature is 0; Qwen 72B uses official defaults, while the remaining models use temperature 0.7, Top P 50, and Top K 0.95.The passage notes that gpt-3.5-turbo-1106 may still produce random outputs at temperature 0.Source: §D.1 · p. 18
• The experiments sample 50 MMLU questions across six domains, 50 MATH cases from Levels 3–5, and 50 Chess Move Validity cases.MMLU domain counts follow an 8:8:8:8:9:9 ratio, and MATH levels follow a 22:22:6 ratio.Source: §D.1 · pp. 18, 20

D.2 Experimental Evaluation

• Society answers are selected by majority vote, with non-unanimous responses counted as errors, then matched to ground truth using rules for format deviations and equivalent answers.Responses containing multiple answers that do not follow the prompts are disregarded; a small number of values may fall outside the matching rules.Source: §D.2 · p. 20

D.3 Illustration of Multi-Agent Collaboration

• In problem-solving, conformity in multi-agent collaboration can be beneficial by changing an answer from wrong to correct, or harmful by changing it from correct to wrong.Source: §D.3 · p. 20
• Figures 9 and 10 present detailed multi-agent conversations illustrating the conformity phenomenon shown in Figure 8.The examples accompany the finding that conformity can produce either beneficial or harmful answer changes.Source: §D.3 · p. 20

E Further Analysis on Machine Social Collaboration (Backbone: ChatGPT)

• Word-cloud and answer-change analyses describe an indistinctive performance impact for three-agent societies, comparing S1’s overconfident agents with S4’s easy-going agents.The word clouds use the top 50 words selected from a manually curated list of task-relevant, high-frequency words; the figures also show proportions of agents keeping answers.Source: E Further Analysis on Machine Social… · p. 20, Figure 11 · p. 23, Figure 12 · p. 23
• Answer changes during a three-agent task in the all-easy-going S4 society illustrate conformity in multi-agent collaboration.Figure 8 tracks answer changes during the process of solving a task with ChatGPT.Source: Figure 8 · p. 21
• Collaborative-strategy p-values were below 0.05; society p-values were below 0.05 in 17 out of 24 gpt-3.5-turbo-1106 cases but above 0.05 in 23 out of 24 July gpt-3.5-turbo cases, supporting stronger strategy influence.The analysis uses ChatGPT with gpt-3.5-turbo-1106 and gpt-3.5-turbo employed between July 10 and July 23, 2023.Source: E Further Analysis on Machine Social… · p. 20, Table 10 · p. 24, Table 9 · p. 24
• Subjects and task difficulty show varying sensitivity to collaborative strategies.The paper presents this analysis using radar maps in Figure 13.Source: E Further Analysis on Machine Social… · p. 24

F Analysis on Machine Society Settings (Backbone: ChatGPT)

• Three agents are relatively optimal, and agent count significantly correlates with performance in the ChatGPT societies.The conclusion integrates significance tests with the accuracy results.Source: F Analysis on Machine Society Settings… · p. 24, Table 11 · p. 25, Figure 15 · p. 27
• Three collaboration rounds are relatively optimal for performance and token cost, although round effects depend significantly on the collaborative strategy.Increasing rounds from 3 to 10 caused performance fluctuations without clear outperformance, while consuming more tokens.Source: F Analysis on Machine Society Settings… · p. 25, Table 12 · p. 25, Figure 18 · p. 29
• Debate tends to reduce consensus clusters, reflection can increase them, and the examined strategies show substantial fluctuations and low answer consistency.An extra debate round was not expected to improve performance, supporting early stopping inspired by Byzantine Consensus theory and implemented in Liu et al. (2023). Continuous reflection in p0p0p0p0 on Chess Move Validity gradually increases clusters with smaller fluctuations; strategies with p-values > 0.05 often show higher answer consistency.Source: F Analysis on Machine Society Settings… · pp. 25, 29
• Consistent thinking patterns have pronounced effects on Chess Move Validity, while their influence is less significant on MMLU and MATH.The analysis compares societies where all agents use the same thinking pattern in a round with societies using different patterns.Source: F Analysis on Machine Society Settings… · p. 29, Table 13 · p. 29, Figure 20 · p. 29

G.1 Conformity and Consensus Reaching

• Conformity is counted when a debating agent gives the most frequent answer, excluding rounds where all answers differ; consensus is the number of distinct answers per round.Conformity is also classified by whether answers are correct before and after it occurs.Source: §G.1 · pp. 29–30

G.2 Group Dynamics

• Group dynamics are analyzed by majority-voting across four answer rounds and grouping selected correctness sequences as correcting mistakes, changing correct answers, or wavering answers.Wavering answers involve self-contradiction and resemble model hallucination (Rawte et al., 2023; Ji et al., 2023; Luo et al., 2024).Source: §G.2 · p. 30

(1) Debate-initial/dominant collaborative

• Debate-initial or debate-dominant strategies are generally more effective and mostly outperform self-consistency, while reflection is more likely than debate to produce wavering answers.The more effective strategies cost more tokens; comparisons of strategy patterns also indicate greater wavering with reflection than with debate.Source: Figure 21 · p. 30, (1) Debate-initial/dominant… · p. 31

H Analysis on Different Backbone LLMs

• The experiments were also implemented with LlaMA2 Chat 13B, LlaMA2 Chat 70B, Qwen 72B, and Mixtral 8×7B.These open-resource backbones were included to broaden the generality of the findings.Source: H Analysis on Different Backbone LLMs · p. 32

H.1 LlaMA2 Chat 13B

• LlaMA2 Chat 13B analyses also vary agent counts, collaboration rounds, and whether agents use the same or hybrid thinking patterns in a round.These analyses are presented in Figures 25–27, with significance tests in Tables 17–19.Source: §H.1 · p. 32, Figure 25 · p. 35, Figure 26 · p. 35
• The LlaMA2 13B social-psychology analyses examine correctness changes under conformity, consensus-cluster counts, and answer-changing behaviors across strategies.Figures 28–30 cover conformity, three-agent consensus clusters across collaboration rounds, and behavioral features across societies.Source: §H.1 · p. 32, Figure 28 · p. 37, Figure 29 · p. 37
• For LlaMA2 Chat 13B, Table 14 reports eight strategies across three datasets and societies, with single-agent comparisons shown in Figure 30(a–f).The accompanying analysis covers answer patterns by society and task sensitivity across subjects and difficulty; the supplied passages do not state which strategy performs best.Source: §H.1 · p. 32, Table 14 · p. 33, Figure 30 · p. 37

H.2 LlaMA2 Chat 70B

• LlaMA2 Chat 70B analyses also track correctness changes under conformity, consensus clusters, and answer-changing behaviors across collaborative strategies.The cited figures summarize results across societies and include token cost and accuracy before and after three-round collaboration.Source: §H.2 · p. 38, Figure 37 · p. 43, Figure 38 · p. 43
• For LlaMA2 Chat 70B, Table 20 reports eight strategies across three datasets and societies, while the analyses also cover agent counts, collaboration rounds, and alternative within-round thinking patterns.The passages identify corresponding significance tests and performance figures but do not report comparative values.Source: §H.2 · p. 38, Table 20 · p. 39, Figure 36 · p. 42

H.3 Qwen 72B

• Qwen 72B analyses also cover conformity-related correctness changes, consensus clusters across rounds and society sizes, and group answer-changing behaviors.Figures 45–46 address consensus with two to ten agents, while Figure 53 summarizes behavioral features, token cost, and accuracy across societies.Source: §H.3 · p. 44, Figure 45 · p. 49, Figure 46 · p. 49
• For Qwen 72B, the appendix examines eight strategies across three datasets and societies, varying agent numbers and rounds, and comparing same-pattern with hybrid within-round strategies.The agent-number analyses include societies of two to ten agents; the round analyses cover three to ten rounds in society S2 on MMLU, MATH, and Chess Move Validity.Source: §H.3 · p. 44, Table 26 · p. 45, Figure 50 · p. 51

H.4 Mixtral 8×7B

• For Mixtral 8×7B, tasks with different subjects and difficulty display varying sensitivity to collaborative strategies, with analyses presented alongside strategy and society results in Tables 32–34 and radar maps in Figure 56.Source: §H.4 · p. 53
• For Mixtral 8×7B, the appendix reports performance across 2–10-agent societies and collaboration rounds, compares same or hybrid thinking strategies, and examines conformity, three-agent consensus clusters, and answer-changing group dynamics.Source: §H.4 · p. 53

I Assessing the Effectiveness of Prompts

• Overconfident agents maintained their viewpoint even when their answer was incorrect, supporting the prompt design’s effectiveness.The prompt check assessed behavioral descriptions, role-play responses, and an “Ask Again” question; the model accepted the prompts without refusal and advocated confidently presenting reasons when agents disagree.Source: I Assessing the Effectiveness of Prompts · p. 62