PaperlayerView exact revision on arXiv

Source-linked AI summary

Better Zero-Shot Reasoning with Role-Play Prompting

Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, Xiaohang Dong

arXiv:2308.07702v2cs.CL

TL;DR

LLM role-playing has been widely used for interaction, but its effect on reasoning remains underexplored. This paper introduces a two-stage zero-shot role-play prompting method and evaluates it across 12 reasoning benchmarks, finding better performance than standard zero-shot and Zero-Shot-CoT on most datasets.

  • Problem

    The influence of LLM role-playing on core reasoning abilities remains underexplored despite its broader use in interaction and selected NLP tasks.

  • Method

    The paper proposes a two-stage role-play prompting methodology for enhancing LLM reasoning under the zero-shot setting.

  • Results

    Role-play prompting outperforms standard zero-shot prompting on 10 of 12 datasets and outperforms Zero-Shot-CoT on 9 of 12 datasets.

  • Takeaways & Limitations

    Role-play prompting can serve as an implicit and effective CoT trigger for improving reasoning outcomes.

  • Takeaways & Limitations

    Role-setting and role-feedback prompts are manually designed and sampled, making prompt construction time-consuming and not always optimal.

Abstract

from arXiv · show

Modern large language models (LLMs) exhibit a remarkable capacity for role-playing, enabling them to embody not only human characters but also non-human entities. This versatility allows them to simulate complex human-like interactions and behaviors within various contexts, as well as to emulate specific objects or systems. While these capabilities have enhanced user engagement and introduced novel modes of interaction, the influence of role-playing on LLMs' reasoning abilities remains underexplored. In this study, we introduce a strategically designed role-play prompting methodology and assess its performance under the zero-shot setting across twelve diverse reasoning benchmarks. Our empirical results illustrate that role-play prompting consistently surpasses the standard zero-shot approach across most datasets. Notably, in experiments conducted using ChatGPT, accuracy on AQuA rises from 53.5% to 63.8%, and on Last Letter from 23.8% to 84.2%.Upon further comparison with the Zero-Shot-CoT technique, which prompts the model to "think step by step", our study demonstrates that role-play prompting acts as a more effective trigger for the CoT process. This highlights its potential to augment the reasoning capabilities of LLMs. We release our code at https://github.com/NKU-HLT/Role-Play-Prompting.

1 Introduction

The paper investigates whether role-playing can improve LLM reasoning and introduces a two-stage zero-shot role-play prompting method. It evaluates this approach against standard zero-shot and Zero-Shot-CoT prompting, proposing role-play as an implicit CoT trigger.

  • Motivation: Role-playing provides LLMs with contextual personas and has improved performance on selected NLP tasks, motivating investigation of its effect on core reasoning abilities.Examples include judging text summarization as a judge and processing Linux commands as a Linux terminal.
  • Research Question: The study asks whether assigning task-relevant roles, such as a math teacher, can enhance ChatGPT’s ability to solve reasoning problems.This question extends prior successes of role-playing beyond conversational and task-specific applications.
  • Contribution: The paper frames its work as a first systematic investigation of role-play prompting for reasoning tasks and an initial basis for further study.It notes that scholarly research on this intersection has been sparse.
  • Method: The proposed role-play prompting methodology uses a two-stage framework to enhance zero-shot reasoning capabilities.The paper presents this as a novel contribution and, to the authors’ knowledge, the first attempt to improve LLM reasoning with role-play prompting.
  • Evaluation: The method is evaluated on 12 reasoning benchmarks and compared with standard zero-shot, Zero-Shot-CoT, and Few-Shot-CoT baselines.The evaluation is intended to assess both effectiveness and prompt-design factors.
  • Interpretation: Role-play prompting is proposed as an implicit CoT trigger that can explain its observed reasoning improvements.The paper distinguishes this from Zero-Shot-CoT, which explicitly appends “Let’s think step by step.”

2 Related Work

Prior work studies role-playing for interaction, evaluation, and role simulation, while reasoning research develops chain-of-thought methods. This paper connects these lines by applying LLM role-playing to reasoning tasks.

  • Role-Playing Abilities: LLMs have been used as role-playing conversation models, summarization evaluators, and role simulators for studying human-like capabilities.These studies focus on interaction, evaluation, and simulation rather than reasoning benchmarks.
  • Role-Playing Abilities: Scholarly investigation of LLM role-playing capacities has remained limited despite growing interest.The paper positions its work as the first application of role-playing abilities to reasoning tasks.
  • Reasoning Abilities: Chain-of-thought prompting improves LLM reasoning by providing or eliciting intermediate reasoning steps.Follow-up methods include few-shot and zero-shot variants.
  • Reasoning Abilities: Few-shot reasoning methods diversify or structure reasoning through techniques such as self-consistency, least-to-most prompting, self-refinement, active prompting, and Tree-of-Thought.These methods sample paths, decompose problems, refine outputs, select uncertain examples, or search reasoning trees.
  • Reasoning Abilities: The paper situates role-play prompting alongside zero-shot chain-of-thought approaches as an alternative way to stimulate reasoning.Figure 2 presents the proposed two-stage role-play framework.

3 Role-Play Prompting

Role-Play Prompting uses a two-stage dialogue that first establishes and selects an immersive role feedback, then applies it to reasoning questions. The procedure combines task-advantaged role design with sampled feedback to generate answers in a single API invocation.

  • 3 Role-Play Prompting: Stage 1 samples multiple role-feedback prompts, while Stage 2 selects the optimal feedback for answer generation.
  • 3 Role-Play Prompting: The method replaces single-turn role assignment with a two-round dialogue that deepens the model’s role framing before question answering.
  • 3.2 Question Answering: The method fixes the most representative and immersive first-round response, then concatenates the first-round interaction with each reasoning question.
  • 3.1 Prompt Construction: The role-setting prompt defines the task-specific persona that the LLM should maintain throughout the dialogue.
  • 3.1 Prompt Construction: The role-feedback prompt is a sampled model acknowledgment intended to anchor the LLM more firmly within the assigned role.
  • 3.1 Prompt Construction: Role selection favors personas with a natural advantage for the target task and descriptions that emphasize that advantage.
  • 3.2 Question Answering: Each question is answered using the role-setting and role-feedback prompts, enabling answer generation with a single API invocation.

4 Experiments

Experiments evaluate Role-Play Prompting across twelve reasoning datasets and compare it with standard zero-shot, Zero-Shot-CoT, and Few-Shot-CoT baselines. The method improves performance across diverse models, scales, prompt designs, and role choices.

  • Evaluation covers 12 datasets spanning arithmetic, commonsense reasoning, and other reasoning task categories, using accuracy as the metric.
  • Role-Play Prompting outperforms standard zero-shot on 10 of 12 datasets and matches it on SingleEq and MultiArith.
  • Role-Play Prompting outperforms Zero-Shot-CoT on 9 of 12 datasets and improves tasks where ChatGPT struggles to generate CoT spontaneously.
  • 4.4 Impact of Prompt Design: A two-round prompt with complementary role descriptions is recommended after prompt-structure experiments on AQuA, Letter, and Coin.
  • 4.4 Impact of Prompt Design: Advantaged roles perform best on AQuA and SVAMP, irrelevant roles follow, and disadvantaged roles underperform the zero-shot baseline.
  • 4.5 Experiments on More LLMs: Role-Play Prompting exceeds zero-shot performance in open-source conversational LLMs, including Llama 2-Chat and Vicuna, across multiple datasets.
  • 4.5 Experiments on More LLMs: All tested Llama 2-Chat sizes—7B, 13B, and 70B—improve with Role-Play Prompting on GSM8K, MultiArith, and Letter.

5 Conclusion

The paper proposes a two-stage zero-shot role-play prompting methodology to improve LLM reasoning. Across twelve benchmarks, it outperforms standard zero-shot prompting and Zero-Shot-CoT on most datasets, positioning role-play as an implicit CoT trigger and motivating further study.

  • The proposed methodology is a two-stage zero-shot role-play prompting framework aimed at enhancing LLM reasoning capabilities.
  • Across twelve widely used benchmarks, Role-Play Prompting outperforms both standard zero-shot prompting and Zero-Shot-CoT on most datasets.
  • The results highlight role-play prompting as an implicit and effective CoT trigger associated with enhanced reasoning outcomes.
  • The study provides initial groundwork for deeper investigation into the intersection of role-playing and reasoning in LLMs.

Limitations

Role-play prompting depends on manually designed role-setting and role-feedback prompts. This design process is time-consuming and may not consistently produce optimal prompts.

  • Manual role-setting and role-feedback prompt design is central to the approach.
  • The manual design and sampling process is time-consuming.
  • Manually designed prompts may not always guarantee optimal results.

A Implementation Details

The implementation constructs a conversational API request from role-setting, role-feedback, and question messages, then performs a second model call to extract answers. Standard zero-shot, Zero-Shot-CoT, and Few-Shot-CoT provide comparison baselines.

  • The API conversation sends the role-setting prompt, role-feedback prompt, and target question as sequential messages.
  • The implementation uses ChatGPT with temperature 0 and a maximum of 512 generated tokens.
  • Answer extraction concatenates each question, generated answer, and answer trigger before a follow-up model input.
  • The baselines are standard zero-shot prompting, Zero-Shot-CoT with “Let’s think step by step,” and Few-Shot-CoT with reasoning exemplars.

A.4 Experiments on More LLMs

The study evaluates role-play prompting on additional conversational LLMs and multiple reasoning datasets, while also examining answer-trigger formats and self-consistency decoding. With self-consistency, role-play prompting continues to outperform the zero-shot baseline.

  • Experiments use Llama 2-Chat and Vicuna on GSM8K, Multiarith, SVAMP, CSQA, and Letter.
  • Llama 2-Chat often declines dataset questions because of safety concerns imposed by RLHF.
  • Answer-trigger sentences are provided for various answer formats.
  • With self-consistency, role-play prompting still consistently outperforms the zero-shot baseline across multiple datasets.The evaluation combines methods with sampled reasoning paths and majority-vote answer selection.

B.3 Ablation Study on Letter, Coin Datasets

Additional ablations on Last Letter and related datasets examine prompt immersion and prompt length. The results support a two-round prompt structure with complementary descriptions, while also testing whether gains arise from length alone.

  • Ablation Study on Letter, Coin Datasets: Four prompt groups with progressively increasing immersion levels are evaluated on Letter and Coin datasets.
  • Ablation Study on Letter, Coin Datasets: The two-round prompt structure with complementary descriptions enhances model immersion and demonstrates effectiveness.
  • Ablation Study on Letter, Coin Datasets: Additional Letter experiments replace role-feedback prompts with generic responses of varying lengths that lack immersion.
  • Ablation Study on Letter, Coin Datasets: Table 10 compares prompt designs with a fixed teacher role on the Last Letter dataset.

B.5 Detailed Results of Model Scale Study

The study examines how model scale affects role-play prompting by evaluating Llama 2-Chat models of 7B, 13B, and 70B parameters on three reasoning datasets.

  • 7B, 13B, and 70B Llama 2-Chat models are evaluated to examine the impact of model scale.The experiment uses the Llama 2-Chat series at three parameter sizes.
  • GSM8K, MultiArith, and Letter are the datasets used in the model-scale experiment.

C Dataset Deatils

The study uses twelve reasoning datasets spanning arithmetic and other reasoning tasks, and examines role selection with eight roles and structured prompts. Dataset and prompt details are documented in Tables 14 and 15, while prompt-design variations are reported in Tables 11–13.

  • Datasets: Twelve datasets span four categories, including six arithmetic datasets and additional BIG-bench reasoning tasks.The arithmetic datasets are MultiArith, GSM8K, AddSub, AQUA-RAT, SingleEq, and SVAMP.
  • Datasets: Arithmetic questions contain a scenario and require reasoning based on mathematical knowledge.
  • Datasets: The experiments use a released version of datasets originally created following Wei et al. (2022), because the originals were unavailable.
  • Datasets: Date Understanding involves date calculations, whereas Tracking Shuffled Objects asks for final object ownership after exchanges.
  • Role prompts: Eight different roles are designed to investigate role selection’s impact on role-play prompting.
  • Prompt analyses: Table 11 compares prompt designs with a fixed coin role on the Coin Flip dataset.Gray shading marks content added relative to the previous prompt.
  • Role prompts: Table 15 pairs each role-setting prompt with a role-feedback prompt.The role-setting prompt appears in the upper cell and the role-feedback prompt in the lower cell.
  • Dataset documentation: Table 14 records each dataset’s question count and average question length.Nq denotes the number of questions, and Lq denotes average question words.
Loading 2308.07702v2…