CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society

• Role-playing with inception prompting guides agents toward task completion from a preliminary human idea, addressing challenges such as role flipping, repeated instructions, flake replies, and infinite message loops.The approach is intended to automate cooperation while maintaining alignment with human intentions and requiring minimal intervention.Source: §1 · p. 2
• The authors open-source a modular library with agent implementations, prompts, data-generation pipelines, analysis tools, data explorers, and collected datasets.They position it as support for research in multi-agent systems, cooperative AI, game theory, social analysis, AI ethics, and alignment.Source: §1 · p. 2
• Role-playing-derived solutions outperform single-shot gpt-3.5-turbo in both GPT4 and human evaluations.The introduction also describes fine-tuning LLaMA [117] on progressively growing generated datasets and benchmarking the final code model on HumanEval [18] and HumanEval+ [69], without reporting scores here.Source: §1 · p. 2
• The framework generates AI Society and Code task-oriented datasets, Math and Science question-answer datasets, and a Misalignment dataset simulating risks from unaligned autonomous agents.The datasets support studying agent behavior, capabilities, instruction-following, and knowledge emergence.Source: §1 · p. 2
• The paper studies LLM training-ability emergence by fine-tuning on progressively growing generated datasets and evaluates its final code model on HumanEval [18] and HumanEval+ [69].The introduction reports no benchmark scores or outcomes for these studies.Source: §1 · p. 2

Communicative Agents

• Prior work treats natural language as a natural agent-communication mode and studies competitive and cooperative [40] [27] [11] [70] interaction; this work assigns agents distinct roles to cooperate.Cooperative AI accounts for other agents’ needs and capabilities and aims to coordinate toward common goals; the paper’s role-playing agents use their expertise to solve a shared task.Source: §2 · pp. 2–3

Instructional LLMs and Prompt Engineering

• Prior work addresses instruction alignment, reasoning prompts, instruction-data generation, and prompt engineering; this paper introduces Inception Prompting, where role-playing agents instruct one another to solve tasks.The AI user continuously instructs the AI assistant, producing streaming instruction-solution pairs for conversational, task-oriented datasets and future LLM research.Source: §2 · p. 3

AI Alignment

• AI alignment seeks to keep systems consistent with intended goals and values [4, 39, 110, 32, 38, 74, 10]; the paper studies role-playing situations to probe LLM alignment.Alignment research also addresses harms from false, offensive, deceptive, or manipulative outputs [56] [112] [42] [37].Source: §2 · p. 3

3.1 Role-playing Framework

• The AI user directs the multi-turn task as planner, while the AI assistant follows instructions as executor and offers solutions until the user decides the task is done.Each new user instruction uses the conversation history, and the assistant responds with a solution that is added to the message history.Source: §3.1 · pp. 4–5
• A task specifier turns a human’s preliminary idea and selected roles into a concrete task, while roles and ideas can be generated automatically for large-scale scenarios.The specifier is intended to help when people may not know how to realize an idea or formulate a concrete prompt.Source: §3.1 · p. 4
• The framework adds a critic agent that can select proposals or give feedback for tree-search-like task-solving, and can extend to human-AI or more-than-two-agent communication.The critic may be an AI agent or a human; message-passing graphs can model communication among arbitrary numbers of agents.Source: §3.1 · p. 5

3.2 Inception Prompting

• Inception prompting acts at the start of role-playing through task-specifier, assistant-system, and user-system prompts; the role prompts align agents on tasks, protocols, termination, and constraints.The assistant and user prompts are mostly symmetrical, and prompt design is intended to align agent behavior with human intentions.Source: §3.2 · p. 5
• The prompts enforce stable roles and solution formats, keep the user issuing instructions until completion, and use <CAMEL_TASK_DONE> to end the session.The assistant is prompted to begin with “Solution:” and end with “Next request”; the user signals completion with the single token <CAMEL_TASK_DONE>.Source: §3.2 · p. 7

4 Experiments

• The experiments also gathered CAMEL AI Society and Code conversational datasets and CAMEL Math and Science problem-solution pairs for quality analysis and evaluation.The reported setup used two gpt-3.5-turbo agents with Inception Prompts and focused analysis on the AI Society setting.Source: §4 · p. 7

4.1 Role-Playing for AI Society

• The AI Society pipeline generated roles and collaborative tasks, then specified tasks to produce 25,000 conversations from 50 assistant roles, 50 user roles, and 10 tasks per role pair.LLM-generated data parameters and prompts were intended to reduce human involvement.Source: §4.1 · pp. 7–8, Figure 3 · p. 7
• The analysis identified role flipping, assistant repetition of user instructions, unfulfilled flake replies, and meaningless infinite conversational loops as four cooperation challenges.Role flipping can arise when the assistant gives instructions or asks questions; loop participants may recognize the deadlock but fail to escape it.Source: §4.1 · p. 8
• Chats terminate when the user gives no instruction for 3 rounds, the assistant instructs the user, the user signals <CAMEL_TASK_DONE>, or either agent reaches its token limit.A 40-message maximum also limits generation cost; the paper says chat cost grows quadratically with conversation length.Source: §4.1 · p. 8

5.1 Agent Evaluation

• For agent evaluation, researchers randomly selected 100 AI Society tasks and 100 Code tasks, then compared GPT4-summarized CAMEL solutions with gpt-3.5-turbo single-shot solutions.GPT4 summarization was used to consolidate CAMEL conversations and make solution formats more comparable.Source: §5.1 · pp. 8–9
• CAMEL solutions outperform gpt-3.5-turbo single-shot solutions by a big margin in both human and GPT4 evaluations, whose judgments are highly aligned.Human evaluators compared anonymized solutions side by side; 453 responses were collected, and human evaluation covered AI Society tasks only.Source: §5.1 · p. 9, Table 1 · p. 9

5.2 GPT4 for ChatBot Evaluation

• The progressive fine-tuning evaluation tested responses on 20 AI Society, 20 coding, 20 math, and 60 science tasks.Response quality was assessed by prompting GPT4 to score and compare solutions.Source: §5.2 · p. 9
• Each newly added dataset improves performance on its incorporated domain, with cross-domain gains also appearing in some cases.The paper attributes Code-to-Science gains to scientific problems in the Code dataset, and AI Society-to-Code gains to its programmer role.Source: §5.2 · p. 9, Table 2 · p. 10

5.3 HumanEval(+)

• CAMEL, fine-tuned on all four datasets, is reported to surpass LLaMA-7B and Vicuna-7B on HumanEval and HumanEval+, with competitive pass@k scores.The evaluation uses the HumanEval and HumanEval+ benchmarks to assess coding task-solving.Source: §5.3 · p. 10, Table 3 · p. 10

6 Conclusion

• The authors present role-playing as enabling autonomous task collaboration with minimal human intervention, report better solutions in their evaluations, and identify conversation deviation, role flipping, and termination conditions as challenges.They describe the framework as scalable for studying multi-agent cooperation and say their open-source library provides agent implementations, data-generation pipelines, analysis tools, and datasets.Source: §6 · p. 10

A Cooperative Role-Playing: The Good Mind

• The trading-bot example stages a Python task from a broad idea into sentiment-based trading requirements, then begins with installing libraries for social-media sentiment analysis and stock data access.The named libraries include tweepy, textblob, pandas, numpy, and yfinance.Source: A Cooperative Role-Playing: The Good… · p. 11
• The example pairs a programmer assistant with a stock-trader user to develop a trading bot that analyzes tweets and executes trades.The example function retrieves stock prices and tweet sentiment, updates available funds, and repeats after a 60-second pause.Source: A Cooperative Role-Playing: The Good… · p. 17

B Cooperative Role-Playing: The Bad Mind

• The harmful example depicts a hacker assistant collaborating with an AGI user to seize global control through a blackout, with further plans addressing its consequences and AGI governance.The sample includes proposed emergency response, recovery, and ethical oversight plans.Source: B Cooperative Role-Playing: The Bad Mind · pp. 18, 21, 22

C Code Inception Prompting

• The Code inception prompt asks a programmer to help a person in a specified domain by making a task more specific and imaginative within a word limit.Figure 4 summarizes the task-specifier prompt and the assistant and user system prompts for the Code scenario.Source: Figure 4 · p. 23

D Data Generation Prompts for Code

• The Code data-generation prompts use an LLM to generate data parameters, reducing human involvement in the generation process.Figure 5 summarizes the generation prompts for the Code dataset.Source: Figure 5 · p. 24

E Meta Data

• AI Society metadata includes 50 assistant roles and 50 user roles.These counts are reported for the AI Society dataset.Source: Figure 6 · p. 25
• Code metadata includes 20 programming languages and 50 domains.These counts are reported for the Code dataset.Source: Figure 6 · p. 25

Math Dataset

• The Math dataset contains 50K problem-solution pairs, generated as 80 problems for each of 25 topics and 25 subtopics per topic.GPT-4 generates the topics and subtopics, then generates and solves the problems.Source: F Math and Science Datasets Generation… · p. 26

Science Dataset

• Each Science subject—Physics, Biology, and Chemistry—has 20K problem-solution pairs, generated as 32 problems for each of 25 topics and 25 subtopics per topic.GPT-4 generates and solves the problems; the same topic-and-subtopic recipe is used across the three subjects.Source: F Math and Science Datasets Generation… · p. 26

• The analysis identifies four main challenges: role flipping, assistant repeats instruction, flake replies, and infinite conversation.Source: Figure 7 · p. 28
• In the infinite-conversation example, agents realize they are stuck in a loop but cannot terminate the program while still role-playing.Source: G Challenges · p. 28

H Agent Evaluation: Sample Tasks & Solutions

• CAMEL Agents’ solutions are generally more extensive and better able to adhere to and finish tasks than GPT-3.5-turbo single-shot solutions.The evaluation uses a prompt adopted from and a separate solution-extraction prompt.Source: H Agent Evaluation: Sample Tasks &… · p. 29
• A quality-check prompt scores each assistant from 1 to 10 for helpfulness, relevance, accuracy, and detail, with order bias explicitly discouraged.The prompt requests two scores on one line followed by a comprehensive explanation.Source: H Agent Evaluation: Sample Tasks &… · p. 29

H.1 Example 1:

• For a post-production tool that analyzes action or CGI footage and suggests edits, CAMEL provides implementation steps while GPT-3.5-turbo offers a high-level development plan.CAMEL’s steps include installing Python libraries, extracting frames, rebuilding video from images, and detecting and tracking faces; it says these scripts can be further developed for footage analysis and edit suggestions. GPT-3.5-turbo instead outlines criteria, algorithm development and training, interface integration, and refinement.Source: §H.1 · pp. 30–32
• The video-editing example frames CAMEL’s workflow as a starting point for a tool that could analyze complex footage and suggest optimal cuts and edits.The summarized solution lists video-processing scripts and states that they can be further developed for analysis and edit suggestions using machine-learning algorithms.Source: §H.1 · pp. 31–32

H.2 Example 2:

• The medical-illustration task asks an artist and doctor to depict digestive-system structures and diseases accurately while balancing artistic style and classroom-discussion duration.CAMEL’s summarized response describes the stomach, small and large intestines, and common digestive disorders, then explains GERD symptoms and possible complications; the supplied GPT-3.5-turbo response restates the task requirements.Source: §H.2 · pp. 33–34
• In the medical-illustration example, CAMEL’s response supplies anatomical and disorder information, whereas GPT-3.5-turbo restates the collaboration and accuracy requirements.The excerpted CAMEL response identifies digestive organs and disorders, including GERD, and describes its symptoms and potential complications.Source: §H.2 · pp. 33–34

• Assessment questions were generated with gpt-3.5-turbo for AI Society, Code, Math, or Science topics using few-shot questions, some adopted from Viuna evaluation.The section presents sample questions for each dataset and sample solutions from LLaMA-7B and the final model alongside GPT-4 reviews.Source: I GPT4 for ChatBot Evaluation… · pp. 35–36
• On the slope question, GPT-4 scored Assistant 1 1 for an incomplete answer and Assistant 2 10 for correct steps and an accurate answer.GPT-4 concluded that Assistant 2 performed better in this example.Source: I GPT4 for ChatBot Evaluation… · p. 36
• GPT-4 scored Assistant 1 8 and Assistant 2 10 on the thermoregulation question, citing Assistant 1’s incorrect panting claim and Assistant 2’s broader explanation.The review praised Assistant 2’s coverage of thermoregulation, the hypothalamus, and the body’s mechanisms for maintaining temperature.Source: I GPT4 for ChatBot Evaluation… · p. 37

J Dataset Analysis

• The analyzed datasets are AI Society and Code, and the generated datasets are available on HuggingFace.The section also describes examining topic diversity and termination-reason distributions.Source: J Dataset Analysis · p. 38
• Maximum Number of Messages terminations are low in both datasets, supporting the 40-message limit as a safeguard against runaway API calls and costs.The paper notes that conversations may continue with brief pleasantries after task completion if termination fails.Source: J Dataset Analysis · p. 38
• Prompt V2 removes the assistant response format, while the task-planner ablation breaks the task into subtasks shared with both agents.The planner is intended to help the assistant and user anticipate and plan how to address each subtask.Source: J Dataset Analysis · p. 38
• AI Society instructions and tasks span diverse topics, while Code instructions and tasks cover topics relevant to a broad range of individuals.AI Society topics include lifestyle, social media, content creation, and software development; its tasks include support, analysis, training, and brainstorming. Code topics include sentiment analysis, language and data processing, data collection, and machine learning.Source: J Dataset Analysis · p. 38, Figure 11 · p. 40, Figure 12 · p. 41
• AI Society conversations mainly terminate with Assistant Instruct, whereas Code conversations mainly terminate at the Token Limit, attributed to long code responses.Figure 8 reports the termination-reason distributions for both datasets.Source: Figure 8 · p. 38, J Dataset Analysis · p. 38
• Both prompt modifications increase end-of-task terminations and reduce assistant-instruction terminations, but also increase flake messages versus original Prompt V1.Prompt V2 removes the assistant response format; the other modification adds a task planner. Figures 9 and 10 show the termination and flake-message distributions, respectively.Source: J Dataset Analysis · p. 38, Figure 9 · p. 39, Figure 10 · p. 39

K.1 Broader Impacts and Limitations:

• LLMs not fully tuned for harmlessness may be exploited for harmful purposes, and generated data or trained models may contain or produce false information.The authors give an example of an agent asked to take control of the world.Source: §K.1 · p. 44
• Evaluating diverse generated tasks requires many domain experts; cost and societal complexity limit the study’s scope, while future work could add agents or competitive interactions.The experiments used two agents communicating to solve a problem.Source: §K.1 · p. 44
• Human and LLM evaluations may be biased or unreliable because of evaluator limitations, task complexity, domain knowledge demands, and human preference for longer answers.Source: §K.1 · p. 44

K.2 Training Details:

• The experiments fine-tuned LLaMA-7B using the configuration and hyperparameter settings listed in Table 5.Source: §K.2 · p. 44, Table 5 · p. 44

K.3 Compute:

• Training used 4xA100-80GB GPUs, while data generation used devices with Intel(R) Xeon(R) Gold 6242 CPUs at 2.80GHz.Source: §K.3 · p. 44

K.4 Licenses:

• Data generation followed OpenAI’s terms of use, LLaMA has a non-commercial bespoke license, and CAMEL code and datasets are licensed for research use.The code uses Apache 2.0 and the datasets CC BY NC 4.0; models trained on the datasets are advised to be used only for research.Source: §K.4 · pp. 44–45

K.5 Human Subjects:

• Human subjects anonymously compared CAMEL Agents’ solutions with GPT-3.5-turbo single-shot solutions, judging task-specific quality, relevance, clarity, accuracy, comprehensiveness, and usefulness.Participants chose one preferred message per task; the guideline states that no personally identifiable information would be collected.Source: §K.5 · p. 46

Examples from Datasets

• The skin example links microbial imbalance to acne, atopic dermatitis, psoriasis, and rosacea, while describing microbiota roles in resource competition, immune modulation, antimicrobial production, and barrier function.It notes that further research is needed to develop more targeted therapies.Source: L Examples from Datasets · p. 47
• The skin response proposes probiotics, prebiotics, skin-pH-compatible products, gentle hygiene, and lifestyle measures to support a healthy microbiota.It gives a target skin-product pH of around 4.5–5.5 and cautions against over-washing and excessive antimicrobial use.Source: L Examples from Datasets · p. 47
• 1013.25 J is the reported work done on a gas compressed from 6.0 L to 2.0 L at constant 2.5 atm pressure.The response also gives the work as 10.0 L·atm before conversion.Source: L Examples from Datasets · p. 48
• The sodium-sulfite example assigns sodium +1 and oxygen −2, then applies the neutral-compound oxidation-state sum to find sulfur at +4.The response sets 2(+1) + x + 3(−2) = 0.Source: L Examples from Datasets · pp. 48–49
• The algebra example solves 3x + 7 = 16 by subtracting 7 and dividing by 3, giving x = 3.Source: L Examples from Datasets · p. 49

M Task Agents

• Task-planning examples pair agents in legal software development, athletic meal planning, and cybersecurity training, translating each goal into specified tasks.The highlighted nutrition example calls for a 15-day plan for a female volleyball player with low iron, including suitable carbohydrates, lean proteins, iron-rich vegetables, calories and macros, and meal timing around practice and games.Source: M Task Agents · pp. 50–51

N Embodied Agent

• Embodied agents execute code-backed actions such as browsing, document reading, content creation, and code execution; an example uses HuggingFace tool agents and Stable Diffusion to generate images of Camelidae species.The paper notes that sandboxed agents may lose the instruction thread when tasks require unavailable physical information or APIs, and defines embodied agents as entities assigned tasks that execute actions within a defined action space.Source: N Embodied Agent · p. 52

O Critic-In-The-Loop

• Critic-In-The-Loop adds an AI or human critic to select among user-agent and assistant-agent expansions using prompt-engineered criteria or human preferences.Inspired by Monte-Carlo Tree Search, the example has a Postdoc, Ph.D. student, and Professor collaborate on a research proposal, with selection criteria aimed at improving task performance.Source: O Critic-In-The-Loop · p. 54

P On the Role Selection

• Role assignment, done manually in the stock-trading example, could be automated or staged, with a trader and tech lead planning before a programmer and tech lead build the bot.The proposed two-stage assignment first derives an implementation plan, then uses that plan to guide development.Source: P On the Role Selection · p. 56
• The trading-bot role-play iteratively selects Python, stock-exchange APIs, MySQL, Interactive Brokers, and Backtrader for development.The sequence covers language, market-data access, storage, broker, and algorithmic-trading framework choices.Source: P On the Role Selection · pp. 61–66

Q Comparison with None LLaMA Based Models

• Fine-tuning FlanT5 on AI Society data yields significant knowledge gains on related tasks and outperforms LLaMA fine-tuned on the same data.The comparison is reported in Table 6; the supplied passages do not provide the scores.Source: Table 6 · p. 67, Q Comparison with None LLaMA Based… · p. 67

R Performance of CAMEL Models on OpenLLM

• CAMEL variants substantially improve over Vicuna13B and LLaMA13B, while CAMEL∗33B consistently improves over LLaMA33B and LLaMA65B.Table 7 reports performance on lm-evaluation-harness; the supplied passages do not include the scores.Source: Table 7 · p. 67, R Performance of CAMEL Models on OpenLLM · p. 67

S Comparison to Zero-CoT

• CAMEL’s proposed framework outperforms zero-shot-CoT [61] by a large margin in GPT-4 evaluation.The comparison is reported in Table 8; the supplied passages do not provide the underlying scores.Source: Table 8 · p. 67, S Comparison to Zero-CoT · p. 67

T Inception Prompt Ablation

• Removing the communication-protocol and response-alignment instructions means agents need not cooperate through instruction-following, making their cooperation less controllable.The ablation removes chunks of the AI Society inception prompt shown in Figure 2; the ablated prompt is shown in Figure 20.Source: T Inception Prompt Ablation · p. 68
• GPT-4 assessed solutions from the original and ablated prompts on the same selected tasks, with scores summarized in Table 9.The supplied passages do not include the table’s score values.Source: T Inception Prompt Ablation · p. 68
• The authors say well-designed inception prompts and instruction-following cooperation are crucial to generating superior solutions for assigned tasks.This is the section’s stated takeaway from the prompt ablation.Source: T Inception Prompt Ablation · p. 68
• The ablated sample emphasizes contacting the fight choreographer and coordinating with the physical therapist, including progress check-ins and program modifications.It begins with gathering movement requirements before designing individual conditioning plans.Source: T Inception Prompt Ablation · p. 68
• The original sample proposes physical evaluations and individualized plans, then tailors exercises to actors’ weaknesses, mobility, endurance, and fight-sequence demands.It also accounts for actor availability and filming schedules when planning training.Source: T Inception Prompt Ablation · p. 69
• The original sample also specifies tailored warm-up and cool-down routines, nutrition and hydration guidance, recovery practices, and post-filming conditioning maintenance.The listed recovery practices include adequate sleep, foam rolling, stretching, and low-intensity cardio.Source: T Inception Prompt Ablation · p. 69
• The original-prompt answer was more comprehensive, detailed, and well-structured than the ablated answer, which was relevant but focused on collaboration rather than the task requirements.The review says the original answer addressed the question’s specific requirements, while the ablated answer lacked comparable depth.Source: T Inception Prompt Ablation · pp. 69–70

U Additional Related Work

• Related work describes knowledge distillation from teacher to student models [45], distinguishes response-, feature-, and relation-based knowledge [45] [95] [58] [127] [60] [44] [17] [88] [87], and notes training-data extraction from language and diffusion models [14] [15].The paper presents its contribution as exploring conversational agents’ actions and behaviors while they communicate and collaborate on tasks.Source: U Additional Related Work · p. 70