PaperlayerView exact revision on arXiv

Source-linked AI summary

Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration

Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, Heng Ji

arXiv:2307.05300v4cs.AIcs.CL

TL;DR

LLMs face factual-hallucination and slow-thinking challenges on knowledge- and reasoning-intensive tasks, while lacking cognitive synergy through collaboration. SPP has one LLM identify, simulate, and collaborate with multiple personas, and the paper reports reduced hallucination and enhanced reasoning.

  • Problem

    LLMs face factual-hallucination and slow-thinking challenges on knowledge- and reasoning-intensive tasks, unlike humans who can draw on cognitive synergy through collaboration and information integration.

    Source: §1 · p. 1

  • Method

    Solo Performance Prompting (SPP) prompts a single LLM to identify, simulate, and collaborate with multiple personas for general task-solving.

    Source: §1 · p. 2

  • Results

    SPP significantly reduces factual hallucination while enhancing reasoning, with superior results to Standard, CoT, and Self-Refine.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Cognitive synergy emerged only in GPT-4, not in the less capable GPT-3.5-turbo and Llama-13b-chat models tested.

    Source: §1 · p. 3

    Even with a fine-grained persona, answers may remain incorrect, and the extent to which personas enhance domain knowledge remains unclear.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Human intelligence thrives on cognitive synergy, where collaboration among different minds yield superior outcomes compared to isolated individuals. In this work, we propose Solo Performance Prompting (SPP), which transforms a single LLM into a cognitive synergist by engaging in multi-turn self-collaboration with multiple personas. A cognitive synergist is an intelligent agent that collaboratively combines multiple minds' strengths and knowledge to enhance problem-solving in complex tasks. By dynamically identifying and simulating different personas based on task inputs, SPP unleashes the potential of cognitive synergy in LLMs. Our in-depth analysis shows that assigning multiple fine-grained personas in LLMs improves problem-solving abilities compared to using a single or fixed number of personas. We evaluate SPP on three challenging tasks: Trivia Creative Writing, Codenames Collaborative, and Logic Grid Puzzle, encompassing both knowledge-intensive and reasoning-intensive types. Unlike previous works, such as Chain-of-Thought, that solely enhance the reasoning abilities in LLMs, experimental results demonstrate that SPP effectively reduces factual hallucination, and maintains strong reasoning capabilities. Additionally, comparative experiments show that cognitive synergy only emerges in GPT-4 and does not appear in less capable models, such as GPT-3.5-turbo and Llama2-13b-chat, which draws an interesting analogy to human development. Code, data, and prompts can be found at: https://github.com/MikeWangWZHL/Solo-Performance-Prompting.git.

1 Introduction

LLMs can struggle with factual hallucination on knowledge-intensive tasks and lack slow-thinking capabilities, while existing reasoning-focused prompting does not resolve hallucination. The paper proposes Solo Performance Prompting (SPP), which uses multi-persona self-collaboration to address knowledge- and reasoning-intensive tasks.

Introduction

  • Earlier persona-based approaches can use fixed or task-specific personas, require additional fine-tuning, or incur greater inference costs through multiple LLM instances.SPP is motivated as a single-LLM alternative for general task-solving.Source: §1 · p. 2
  • SPP prompts a single LLM to identify, simulate, and collaborate with multiple personas through multi-turn self-collaboration.Personas may be domain experts or target audiences; dynamic identification supports diverse domain knowledge without additional retrieval systems, while collaboration enables self-revision and feedback without additional agents.Source: §1 · p. 2
  • In a creative-writing example, standard prompting misses essential information and produces factual errors, whereas SPP uses expert personas and multi-turn collaboration to produce informative, coherent answers.The example combines information from the Legend of Zelda, Harry Potter movies, and Jay Chou’s albums; the assistant drafts, solicits feedback, and revises.Source: §1 · pp. 2–3
  • Cognitive synergy emerges with SPP in GPT-4, but not in the less capable GPT-3.5-turbo and Llama-13b-chat models.The authors draw an analogy to human development, noting that children typically begin role-playing around age 2 to 3.Source: §1 · p. 3
  • The paper evaluates SPP on Trivia Creative Writing, Codenames Collaborative, and Logic Grid Puzzle, spanning knowledge- and reasoning-intensive domains.The authors describe it as the first zero-shot prompting method to enhance both knowledge and reasoning abilities on GPT-4.Source: §1 · p. 3

2 Solo Performance Prompting

Solo Performance Prompting (SPP) has an LLM solve general tasks through zero-shot, multi-persona self-collaboration. It dynamically identifies participants, has them brainstorm, and uses iterative feedback to refine the leader persona’s answer.

Overall procedure

  • SPP identifies multiple task-relevant participants, including an AI Assistant leader, who collaborates with them through brainstorming and iterative revision.The prompt includes a high-level instruction and two demonstrations of the expected procedure.Source: §2 · pp. 3–4

Persona Identification

  • SPP dynamically identifies personas rather than manually defining them, and reports that GPT-4 can identify accurate, meaningful personas for diverse tasks from two demonstrations.The demonstrations’ tasks do not overlap with the evaluation tasks.Source: §2 · p. 4

Brainstorming

  • Before the initial answer, the AI Assistant leads brainstorming in which other personas share expertise and suggestions, improving the initial solution’s quality.The AI Assistant is designated persona i = 0, while the other personas are indexed i ≥ 1.Source: §2 · p. 4

Multi-Persona Iterative Collaboration

  • The AI Assistant proposes an initial solution, consults each other participant for critique and revision suggestions, and repeats collaboration until participants are satisfied.The final answer can be read out in the user-specified format.Source: §2 · p. 4

Scope and comparison

  • SPP is presented as a pure zero-shot method for general tasks, unlike prior prompting approaches that may be task-specific or require additional mechanisms.The listed mechanisms include searching, external tools, memory, and fine-tuning.Source: §2 · p. 4

3 Experiments

The experiments assess SPP on knowledge- and reasoning-intensive tasks against standard prompting, Chain-of-Thought, and Self-Refine, and examine whether its benefits depend on model capability.

3 Experiments

  • The evaluation spans knowledge-intensive and reasoning-intensive tasks, comparing SPP with Standard Prompting, CoT, and one Self-Refine iteration that uses three times more inferences than SPP.The experiments primarily use GPT-4 and also test GPT-3.5-turbo and Llama2-13b-chat for model-scale and capability effects.Source: §3 · pp. 4–5

3.1 Trivia Creative Writing: A Knowledge-Intensive Task

  • SPP outperformed all baselines on Trivia Creative Writing, with gains of 7% at N = 5 and 10% at N = 10 over Standard Prompting.The task asks models to write coherent stories incorporating trivia answers; scoring measures correct answer mentions across 100 instances for each N.Source: Table 2 · p. 5, §3.1 · p. 5

3.2 Codenames Collaborative: A Knowledge+Reasoning Task

  • On Codenames Collaborative, SPP improved performance by about 5%, while CoT added no gains and Self-Refine had a negative impact.The task assigns the same model sequential Spymaster and Guesser roles, and evaluates the overlap between predicted and target words.Source: Table 2 · p. 5, §3.2 · pp. 5–6

3.3 Logic Grid Puzzle: A Reasoning-Intensive Task

  • On Logic Grid Puzzle, CoT significantly improved over Standard Prompting, and SPP also achieved strong performance.The benchmark contains 200 puzzles requiring multi-step reasoning; performance is measured by house-number answer accuracy.Source: Table 2 · p. 5, §3.3 · pp. 6–7

3.4 The Emergence of Cognitive Synergy

  • SPP’s effectiveness was not seen in smaller, less capable models, while cognitive synergy was fully unleashed only at an instruction-following level like GPT-4.On Llama2, SPP sometimes triggered early termination after participant identification; the paper compares the capability threshold to children beginning role-play around ages 2–3.Source: §3.4 · p. 7

4 Analysis

SPP improves both knowledge-intensive and reasoning-intensive abilities on GPT-4. The analysis finds that its performance depends on task-relevant personas and model capability, while examining persona profiles, fixed personas, and prompt demonstrations.

Overall analysis

  • SPP improves knowledge-intensive and reasoning-intensive tasks without external knowledge bases, and is presented as the first zero-shot prompting method to enhance both on GPT-4.The tasks include Trivia Creative Writing, Codenames Collaborative, and Logic Grid Puzzle.Source: §4 · p. 7

Persona analysis

  • Identified personas correlate with each task: knowledge-intensive tasks elicit more diverse, specific personas, while reasoning-intensive tasks elicit more homogeneous ones.For example, SPP often identifies a “Logic Puzzle Expert” even when the input does not mention logic puzzles.Source: §4 · p. 7

Persona profiles

  • SPP-Profile does not outperform SPP, suggesting a fine-grained persona name may suffice to elicit domain knowledge without a detailed profile.Source: §4 · pp. 7–8

Model capability

  • SPP achieves superior performance with GPT-4, but not GPT-3.5 or Llama2-13b, suggesting cognitive synergy emerges only at GPT-4-level capabilities.Source: Figure 6 · p. 8

Dynamic personas v.s. fixed personas

  • SPP consistently outperforms fixed “AI Assistant” and “Expert” personas; the fixed-persona variant can also stop collaboration before giving a final answer.Examples show fine-grained personas such as “Film Expert” and “Sports Enthusiast” answering correctly where the fixed “Expert” fails.Source: §4 · p. 8

Impact of the demonstrations in SPP prompt

  • Adding a second demonstration for collaboration among more than two personas boosts performance, while SPP remains robust with only the first example.Source: §4 · p. 8

5 Related Work

Related work explores role-playing personas and methods for improving LLM reasoning or factual knowledge, but the described approaches leave gaps in dynamic persona selection, hallucination reduction, or reasoning. SPP uses one LLM to dynamically identify personas and elicit both knowledge and reasoning abilities.

LLMs as role-playing agents

Enhancing reasoning and factual knowledge in LLMs

6 Conclusion

The conclusion presents Solo Performance Prompting (SPP) as eliciting cognitive synergy in powerful LLMs, reducing factual hallucination while enhancing reasoning, and outperforming Standard, CoT, and Self-Refine on proposed tasks.

6 Conclusion

  • SPP reduces factual hallucination while enhancing reasoning, and achieves superior results to Standard, CoT, and Self-Refine on the proposed tasks.The conclusion names Trivia Creative Writing and Codenames Collaborative as examples.Source: §6 · p. 9
  • The conclusion characterizes cognitive synergy as an emergent ability in different LLMs and draws an analogy to human development.Source: §6 · p. 9

Limitations · A Prompts

The paper notes that SPP’s effect on domain knowledge and its input-independent demonstrations remain unresolved, while the appendices describe its prompt design, examples, and full prompt variants.

Limitations

  • Even with a fine-grained persona, answers may be incorrect, and the extent to which personas enhance domain knowledge remains unclear.The authors call for diagnostic experiments and theoretical work to quantify the impact of using a persona.Source: Limitations · p. 10
  • Using the same SPP prompt and two demonstrations for every input may be suboptimal; future work could select demonstrations conditioned on each input.Source: Limitations · p. 10
  • With sufficient computation, SPP could extend to a leader persona coordinating expert agents, enabling richer computation, larger local memory, and more flexible human-computer interaction.The authors suggest these capabilities could be important for real-world applications.Source: Limitations · p. 10

A.1 SPP Prompt Design

  • The prompt’s system principle asks the model to identify participants and collaborate over multiple turns, while its task prefix adds step-by-step instructions, task-specific format guidance, and inputs.Source: §A.1 · p. 12
  • Two demonstrations illustrate reasoning collaboration through a Game of 24 task and knowledge-intensive collaboration through poem writing with diverse constraints and an audience persona.The poem example includes a dedicated ten-year-old-child audience persona; the Game of 24 example uses an AI Assistant and Math Expert.Source: §A.1 · p. 12

A.2 Full Prompts

  • Appendix A.2 presents full prompts for SPP, SPP-Profile, SPP-Fixed-Persona, Chain-of-Thought, and Self-Refine prompting.Source: §A.2 · pp. 12–13
  • Removing the second, multi-persona demonstration leaves SPP fairly robust, while including it boosts performance on all three tasks.Source: §A.1 · p. 12, Figure 9 · p. 13

B Task Details · B.1 Trivia Creative Writing

In Trivia Creative Writing, SPP is notably superior for larger question sets (N≥5), while question ordering has minimal impact. The task’s topic list is generated by prompting GPT-4 for 100 pop-culture nouns.

B.1 Trivia Creative Writing

  • With N≥5 questions, Trivia Creative Writing challenges GPT-4 and SPP is notably superior; at N=1, prompting methods yield similar outcomes.Question ordering has minimal impact on task performance.Source: §B.1 · p. 13
  • The topic list is automatically generated by prompting GPT-4 to provide 100 nouns from pop culture.Source: §B.1 · p. 13

C Inference Configurations

The section specifies the inference settings for GPT-4, GPT-3.5, and Llama2, and reports that GPT-4 generation is fairly consistent across runs, with lower variance for SPP than for Standard and CoT prompting.

C Inference Configurations

  • GPT-4 results used the Azure 2023-3-15-preview API with temperature 0.0 and top_p 1.0, though generation could still be non-deterministic.The temperature was set to 0.0 as the most conservative setting, and top_p to 1.0 to maximize reproducibility.Source: C Inference Configurations · p. 13
  • ∼1% variance characterized GPT-4 generation as fairly consistent, while SPP showed lower variance than Standard and CoT across three Trivia Creative Writing runs.The consistency experiment used N=5 and temperature 0.0 with top_p 1.0.Source: Table 3 · p. 13, C Inference Configurations · p. 13
  • GPT-4 experiments averaged scores across settings with and without the default system message, with full setting-specific results reported in Appendix F.The system message was “You are an AI assistant that helps people find information”; patterns varied across tasks and methods.Source: C Inference Configurations · p. 13
  • GPT-3.5 used the same prompt and hyperparameters as GPT-4 with the system-message setting best for SPP, while Llama2 used Huggingface text-generation with greedy decoding.The GPT-3.5 setting was selected based on SPP’s GPT-4 performance.Source: C Inference Configurations · p. 13

D Additional Qualitative Analysis

The qualitative examples contrast CoT’s plausible plans but factually inaccurate outcomes with SPP’s precise knowledge and detailed, interpretable dialogues. In Codenames Collaborative, these dialogues are described as outperforming CoT.

D Additional Qualitative Analysis

  • CoT can produce plausible Trivia Creative Writing plans, but its outcomes often contain factual inaccuracies and hallucinations, whereas SPP elicits precise knowledge with fine-grained personas.The examples are presented in Figure 11.Source: D Additional Qualitative Analysis · p. 13
  • In Codenames Collaborative, SPP generates detailed, interpretable intermediate dialogues that lead to superior performance compared to CoT.The examples are presented in Figure 12.Source: D Additional Qualitative Analysis · pp. 13–14
  • Figure 13 provides additional qualitative examples comparing Solo Performance Prompting with SPP-Profile.Source: D Additional Qualitative Analysis · p. 14

E Early-termination with SPP-Fixed-Persona · F Full Results · G Usage of AI assistants in writing

SPP-Fixed-Persona can stop before producing a final solution, especially on Codenames Collaborative; removing the system message reduces but does not eliminate the problem. The section also points to full task results and supplementary qualitative examples, prompts, and AI-assistant usage information.

F Full Results

  • Full results for Trivia Creative Writing, Codenames Collaborative, and Logic Grid Puzzle are reported in Tables 5, 6, and 7, respectively.Each table includes inference settings with and without the system message; “average” and “max” refer to mean and maximum scores across the two settings.Source: F Full Results · p. 14, Table 5 · p. 20, Table 6 · p. 22

G Usage of AI assistants in writing

  • The paper states that ChatGPT and GPT-4 were used solely for checking.The supplied passage does not specify what was checked.Source: G Usage of AI assistants in writing · p. 14
  • On Trivia Creative Writing, CoT can produce reasonable plans yet suffer factual errors and hallucination, as illustrated against SPP.Figure 11 gives qualitative examples for N=5.Source: Figure 11 · p. 15

E Early-termination with SPP-Fixed-Persona

  • Removing the system message can reduce—but not eliminate—SPP-Fixed-Persona early termination, which is particularly severe on Codenames Collaborative.Table 4 reports early-termination statistics by task and method; the passages provide no specific values.Source: E Early-termination with… · p. 14, Table 4 · p. 15

G Usage of AI assistants in writing

  • On Codenames Collaborative, SPP’s detailed, multi-perspective intermediate discussions support stronger knowledge selection, integration, and theory-of-mind capabilities than CoT.Figure 12 presents qualitative examples; no numerical comparison is supplied in the passage.Source: Figure 12 · p. 16
  • Trivia examples show dynamically identified fine-grained personas outperforming fixed general personas.The examples contrast personas such as “Film Expert,” “Music Enthusiast,” and “Sports Enthusiast” with the fixed persona “Expert.”Source: Figure 13 · p. 17
  • Supplementary figures provide the full SPP, SPP-Profile, and SPP-Fixed-Persona prompts.Figures 15–17 identify the prompts and their key differences.Source: Figure 15 · p. 19, Figure 16 · p. 20, Figure 17 · p. 21
  • Additional prompt figures cover CoT, self-refine, and generation of the Trivia Creative Writing topic list.These are shown in Figures 18–20.Source: Figure 18 · p. 22, Figure 19 · p. 22, Figure 20 · p. 22

Related papers

Loading 2307.05300v4…