Source-linked AI summary
Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, Heng Ji
TL;DR
Large language models face factual hallucination and limited slow-thinking in knowledge-intensive and reasoning-intensive tasks, unlike collaborative human cognition. Solo Performance Prompting transforms one LLM into multiple dynamically identified personas that self-collaborate, with improvements over Standard, Chain-of-Thought, and Self-Refine; cognitive synergy emerges only in GPT-4.
Problem
Large language models still encounter factual hallucination and limited slow-thinking capabilities in knowledge-intensive and reasoning-intensive tasks, while lacking human-like collaboration and information integration among cognitive processes and individuals.
Method
Solo Performance Prompting prompts a single LLM to identify, simulate, and collaborate with multiple personas in a pure zero-shot, multi-agent interaction.
Results
Solo Performance Prompting significantly reduces factual hallucination while enhancing reasoning, outperforming Standard, Chain-of-Thought, and Self-Refine; cognitive synergy emerges only in GPT-4, not less capable models.
Takeaways & Limitations
The results indicate that a single powerful LLM can exhibit cognitive synergy through multi-persona self-collaboration for general task-solving.
Takeaways & Limitations
Even with a fine-grained persona, answers may remain incorrect, and the extent to which personas enhance domain knowledge remains unclear.
Abstract
from arXiv · showhide
Human intelligence thrives on cognitive synergy, where collaboration among different minds yield superior outcomes compared to isolated individuals. In this work, we propose Solo Performance Prompting (SPP), which transforms a single LLM into a cognitive synergist by engaging in multi-turn self-collaboration with multiple personas. A cognitive synergist is an intelligent agent that collaboratively combines multiple minds' strengths and knowledge to enhance problem-solving in complex tasks. By dynamically identifying and simulating different personas based on task inputs, SPP unleashes the potential of cognitive synergy in LLMs. Our in-depth analysis shows that assigning multiple fine-grained personas in LLMs improves problem-solving abilities compared to using a single or fixed number of personas. We evaluate SPP on three challenging tasks: Trivia Creative Writing, Codenames Collaborative, and Logic Grid Puzzle, encompassing both knowledge-intensive and reasoning-intensive types. Unlike previous works, such as Chain-of-Thought, that solely enhance the reasoning abilities in LLMs, experimental results demonstrate that SPP effectively reduces factual hallucination, and maintains strong reasoning capabilities. Additionally, comparative experiments show that cognitive synergy only emerges in GPT-4 and does not appear in less capable models, such as GPT-3.5-turbo and Llama2-13b-chat, which draws an interesting analogy to human development. Code, data, and prompts can be found at: https://github.com/MikeWangWZHL/Solo-Performance-Prompting.git.
1 Introduction
LLMs face factual hallucination and limited slow-thinking on knowledge- and reasoning-intensive tasks, motivating SPP, which uses dynamically identified personas for self-collaboration. The paper evaluates this approach across three tasks and reports cognitive synergy only in GPT-4.
- LLMs encounter factual hallucination and limited slow-thinking capabilities on knowledge-intensive and reasoning-intensive tasks.
- Previous prompting approaches can require fixed or task-specific personas, additional fine-tuning, or multiple LLM instances that increase inference costs.
- SPP prompts a single LLM to identify, simulate, and collaborate with multiple personas through multi-turn self-collaboration.Personas may represent domain experts or target audiences, while the process supports self-revision and self-feedback without additional agents.
- Cognitive synergy emerges in GPT-4 but not in GPT-3.5-turbo or Llama-13b-chat.The comparison uses models with varying scales and capabilities.
- SPP is evaluated on Trivia Creative Writing, Codenames Collaborative, and Logic Grid Puzzle, covering knowledge- and reasoning-intensive domains.
- The paper analyzes why dynamic, fine-grained personas are needed instead of fixed, coarse-grained personas.
2 Solo Performance Prompting
Solo Performance Prompting (SPP) turns one LLM into a multi-persona, self-collaborating task-solving agent through dynamic persona identification, brainstorming, and iterative refinement.
- Persona Identification: SPP identifies multiple task-relevant participants with specialized personas, including a leader persona called AI Assistant.Personas may represent domain experts or target audiences, and are dynamically generated rather than manually defined.
- Brainstorming: The identified personas brainstorm from their own perspectives before the AI Assistant generates an initial solution.The paper reports that brainstorming improves the quality of the initial solution.
- Multi-Persona Iterative Collaboration: The AI Assistant iteratively consults other participants, who critique the current generation and suggest revisions until a final answer is reached.Intermediate generations are indexed across n iterations, and the final answer follows the user-specified output format.
- Overview: SPP uses multi-persona self-collaboration in a pure zero-shot manner without requiring additional agents, retrieval systems, external tools, memory, or fine-tuning.The comparison with prior prompting methods identifies these mechanisms as requirements of earlier approaches.
3 Experiments
The experiments evaluate SPP across knowledge-intensive and reasoning-intensive tasks using GPT-4, with additional model comparisons and baselines. SPP improves factual performance and collaborative-task results while retaining strong reasoning performance, but its effectiveness is absent in smaller models.
- Evaluation Setup: The evaluation covers Trivia Creative Writing, Codenames Collaborative, and Logic Grid Puzzle, spanning knowledge-intensive and reasoning-intensive domains.The study compares Standard Prompting, Chain-of-Thought, and Self-Refine baselines, primarily using GPT-4.
- Trivia Creative Writing: Trivia Creative Writing uses 100 instances each for N = 5 and N = 10, integrating answers to diverse trivia questions into coherent stories.The benchmark contains 1000 trivia questions extracted from TriviaQA; evaluation counts matching answer mentions rather than subjective story coherence.
- Trivia Creative Writing: SPP significantly outperforms all baselines on Trivia Creative Writing, with relative improvements of 7% for N = 5 and 10% for N = 10.CoT does not outperform Standard Prompting, while Self-Refine provides only marginal improvements; SPP’s advantage grows with more questions.
- Codenames Collaborative: Codenames Collaborative assigns the same GPT-4 sequentially to Spymaster and Guesser roles in a 50-instance task requiring knowledge, reasoning, and theory of mind.Performance is measured by the overlap between predicted and target words, avoiding human annotation.
- Codenames Collaborative: SPP improves Codenames Collaborative results by approximately 5%, whereas CoT adds no positive gains and iterative Self-Refine reduces performance.The reported negative effect of Self-Refine is attributed to changing an initially good response.
- Model Comparison: Cognitive synergy appears in GPT-4 but not in GPT-3.5-turbo or Llama2-13b-chat.On Llama2, SPP can trigger early termination after participant identification, producing exceptionally low performance.
4 Analysis
SPP improves both knowledge-intensive and reasoning-intensive task performance, with benefits tied to dynamically identified personas and GPT-4-level model capability. Analyses also examine persona profiles, fixed personas, and demonstration examples.
- SPP improves both knowledge and reasoning abilities, including knowledge-intensive tasks without external knowledge bases and reasoning-intensive Logic Grid Puzzle.The method is reported to improve Trivia Creative Writing and Codenames Collaborative while maintaining strong performance on Logic Grid Puzzle.
- Automatically identified personas are closely correlated with the particular task.Figure 7a visualizes persona frequencies by task, indicating task-specific persona selection.
- SPP-Profile does not outperform SPP, suggesting that persona names without detailed descriptions may suffice to elicit certain domain knowledge.
- SPP achieves superior performance only with GPT-4, not with GPT-3.5 or Llama2-13b.The analysis attributes this pattern to cognitive synergy emerging only in models with GPT-4-level capabilities.
- SPP consistently outperforms SPP-Fixed-Persona across the evaluated comparisons.The ablation contrasts dynamic task-specific personas with fixed “AI Assistant” and “Expert” personas.
- Adding a second demonstration with more than two personas boosts performance, while SPP remains fairly robust when that example is removed.
5 Related Work
Related work studies persona-based role playing and methods for improving reasoning or factual knowledge in LLMs. SPP is positioned as eliciting both capabilities without external knowledge bases.
- Persona assignment and multi-agent collaboration have been explored, but prior approaches include single or fixed persona assignments.
- Chain-of-Thought and Self-Refinement encourage stepwise solving or iterative revision, but do not necessarily reduce factual hallucination.
- Figure 7b quantitatively analyzes two SPP variants, while Figure 8 qualitatively compares SPP with CoT and SPP-Fixed-Persona.
- Retrieval-augmented LLMs enhance knowledge acquisition but do not improve reasoning abilities.
- SPP is proposed to elicit both knowledge and reasoning abilities, improving factuality while maintaining strong performance on pure-reasoning tasks.
6 Conclusion
SPP uses self-collaboration among personas to improve factual knowledge and reasoning. Its reported advantages include reduced factual hallucination and stronger results than several prompting baselines.
- SPP significantly reduces factual hallucination while enhancing reasoning.
- SPP achieves superior results compared with Standard, CoT, and Self-Refine on newly proposed tasks.
- Cognitive synergy emerges differently across LLMs, drawing an analogy to human development.
Limitations
The paper identifies limitations involving residual factual errors, fixed demonstrations, and the computational demands of a possible multi-agent extension. It also outlines the SPP prompt structure and demonstration design.
- Limitations: Even with a fine-grained persona, SPP can still produce incorrect answers, and the impact of assigning personas remains unquantified.The authors call for diagnostic experiments and theoretical work to measure the effect of having a persona.
- Limitations: Using the same two demonstration examples for every task input may be suboptimal.Conditioning demonstration-example selection on each input is proposed as a future improvement.
- Limitations: A multi-agent cognitive synergist extension would require sufficient computational budget.The proposed setup could use a leader persona and several expert agents in a collaborative cabinet.
- SPP prompt structure: The SPP prompt combines a system principle, two crafted demonstrations, and a task prefix.
- Demonstration design: The demonstrations include a Game of 24 example with an AI Assistant and Math Expert, while the task prefix requests step-by-step collaborative solving.
- Prompt variants: The prompt materials include variants for SPP-Profile, SPP-Fixed-Persona, CoT, and Self-Refine.
- Implementation settings: The same prompt is used for arbitrary tasks, and the GPT-3.5 and Llama2 settings use specified model-specific decoding and prompt configurations.
- Demonstration ablation: Adding a multi-persona demonstration boosts performance across all three tasks, while the prompt remains fairly robust without it.
B Task Details
The Trivia Creative Writing task tests whether models can integrate answers to multiple trivia questions into a coherent story. Experiments focus on settings with five or ten questions, using a 1,000-question benchmark.
- Task difficulty: N≥5 effectively challenges GPT-4, whereas N=1 produces similar outcomes across prompting methods.The study therefore mainly evaluates N=5 and N=10.
- Benchmark construction: The topic list is automatically generated by prompting GPT-4 for 100 pop-culture nouns.The authors additionally curate the list to avoid potentially harmful content.
C Inference Configurations
The experiments use controlled GPT-4 inference settings and assess generation consistency, system-message effects, and results on other model implementations. GPT-4 generation is generally stable under the reported configuration.
- Inference settings: GPT-4 experiments use Azure API version 2023-3-15-preview with temperature 0.0 and top_p 1.0.These settings are intended to maximize reproducibility, although generation can remain nondeterministic.
- Data handling: Content-filter-triggering generations are excluded from the results.The authors describe these as rare API cases.
- Generation consistency: GPT-4 generation has approximately 1% variance across three Trivia Creative Writing runs, with SPP varying less than Standard and CoT prompting.The consistency experiment uses temperature 0.0 and top_p 1.0.
- System-message effects: Results average scores across inference with and without the default system message.The two settings show divergent patterns across tasks and methods.
- Model configurations: GPT-3.5 uses the same prompt and hyperparameters as GPT-4 with the system-message setting selected by GPT-4 performance, while Llama2 uses greedy decoding.These implementation choices define the reported comparisons for the less capable models.
D Additional Qualitative Analysis
Qualitative analyses contrast SPP with CoT and profile-based prompting across trivia and collaborative tasks. They associate SPP with fewer factual errors, richer discussions, and benefits from dynamically specialized personas.
- Trivia Creative Writing: CoT produces plausible task-resolution plans, but final Trivia Creative Writing outputs often contain factual inaccuracies and hallucinations.SPP instead elicits more precise knowledge using fine-grained personas.
- Question analysis: The impact of question number and ordering is analyzed for Trivia Creative Writing.Figure 10 specifically examines N and question ordering.
- Codenames Collaborative: SPP generates detailed, interpretable intermediate dialogues from multiple perspectives in Codenames Collaborative.The qualitative comparison links these discussions to knowledge selection, integration, and theory-of-mind capabilities.
E Early-termination with SPP-Fixed-Persona
The analysis documents early termination mainly for SPP-Fixed-Persona and some less capable-model settings, while comparing prompts, full results, and qualitative examples across tasks. Removing the system message reduces but does not eliminate the issue.
- Early termination: Early termination occurs when generation stops before the final solution as though the model were awaiting external user input.The examples focus on SPP-Fixed-Persona and less capable-model settings.
- Early termination: The problem is especially severe on Codenames Collaborative, producing unexpectedly low performance.Removing the system message largely alleviates but does not fully eliminate early termination.
- Model and method comparison: GPT-4 shows no observed early termination with SPP, SPP-Profile, Standard, or CoT prompting.The reported issue is contrasted with the SPP-Fixed-Persona and Llama2 examples.
- Full results: Full results are reported separately for Trivia Creative Writing, Codenames Collaborative, and Logic Grid Puzzle, including averages and maxima across system-message settings.The tables cover the three tasks and their two inference settings.
- Early-termination statistics: SPP-Fixed-Persona early termination is reduced by removing the default system message but remains possible.The corresponding table summarizes the problem’s statistics.
- Prompt materials: The supplementary materials provide full prompts for SPP, SPP-Profile, SPP-Fixed-Persona, CoT, and Self-Refine.They also include a prompt for generating Trivia Creative Writing topic lists.
- Prompt variants: SPP-Fixed-Persona restricts participants to AI Assistant or Expert, whereas SPP allows task-specific participant identification.SPP-Profile additionally asks for participant expertise or needs.