PaperlayerView exact revision on arXiv

Source-linked AI summary

A Large-Scale Evaluation of Role-Enhanced Zero-Shot Chain-of-Thought Prompting

Ibrahim Khan, Thai Van Nguyen, Yi Xia, Pratch Suntichaikul, Mustafa Can Gursesli, Daniel Fernandez Galeote, Juho Hamari, Ruck Thawonmas

arXiv:2610.29073v1cs.CLcs.AI

TL;DR

Zero-shot reasoning and role-enhanced prompting can be inconsistent, while evidence on role prompting’s scalability and adaptability remains limited. RE-ZS-CoT combines expert roles with zero-shot chain-of-thought prompting and evaluates static and model-selected roles; results vary by task, with static roles strongest on structured arithmetic and dynamic roles often competitive on commonsense and reading tasks.

  • Problem

    Zero-shot prompting methods can perform inconsistently across tasks, and role-enhanced prompting evaluations have not established scalability, adaptability, or effects across reasoning domains.

    Source: §I · p. 1

  • Method

    RE-ZS-CoT integrates expert roles into single-turn zero-shot reasoning, testing predefined roles and roles autonomously selected by each model without task examples.

    Source: §I · p. 1, §III · p. 3, §IV · p. 4

  • Results

    Static roles peak on structured arithmetic while dynamic roles often lead on commonsense and reading tasks; Gemma3-12b rises from 23.2% zero-shot-CoT to 67.3% static on AQuA-RAT.

    Source: A. Overall Results · p. 5, §VII · p. 7

  • Takeaways & Limitations

    The study concludes that domain alignment matters in prompt design and reports role-aware CoT gains without fine-tuning or multi-turn interactions.

    Source: §I · p. 1, §VIII · p. 9

    The study does not conduct paired statistical significance tests for every model-task pair, so smaller single-digit gains on limited-sample datasets should be treated as competitive rather than definitive.

    Source: §VII · p. 8

Abstract

from arXiv · show

This paper proposes Role-Enhanced Zero-Shot Chain-of-Thought (RE-ZS-CoT), a prompting framework that systematically integrates domain-specific expert roles to enhance reasoning in large language models (LLMs) without task-specific examples. We assess two versions of our approach. In the first version, static role selection, we evaluate eight predefined expert roles across arithmetic, commonsense, and reading comprehension benchmarks to determine their impact on reasoning performance. In the second version, dynamic role selection, each model autonomously selects the single most suitable role for each benchmark, enabling adaptive task-role alignment without manual specification. Across 20 LLMs and seven benchmarks, RE-ZS-CoT frequently surpasses zero-shot and zero-shot-CoT baselines, achieving performance improvements of up to 44 percentage points. The results reveal a strong correspondence between static and dynamic versions; the benchmarks where static roles yielded improvements were almost always those where dynamic roles also outperformed baselines, though static selection achieved peak performance. Ablation studies further show that unconventional roles degrade performance, underscoring the necessity of task-role alignment. The framework's scalability is validated across model sizes (2b-14b parameters), with smaller models like Granite3.3-8b achieving remarkable proficiency in arithmetic reasoning. RE-ZS-CoT advances zero-shot prompting by unifying role specialization with chain-of-thought reasoning, offering a practical, resource-efficient solution for improving LLM capabilities across diverse domains.

I. INTRODUCTION

The introduction identifies inconsistent zero-shot reasoning and a gap in scalable, adaptive role-prompting evaluation. It presents RE-ZS-CoT as a two-version framework and reports broad gains alongside task-dependent role effects.

I. INTRODUCTION

  • Zero-shot-CoT and related structured prompting methods can perform inconsistently across tasks, especially when task-specific examples are unavailable.The cited approaches include Plan-and-Solve and Graph of Thoughts,.Source: §I · p. 1
  • Role-enhanced prompting evaluations had not established its scalability, adaptability, or effects across reasoning domains.Prior persona prompting could improve accuracy, though not consistently.Source: §I · p. 1
  • RE-ZS-CoT integrates expert-role specification into single-turn zero-shot reasoning without task-specific examples.The paper evaluates static role selection and adaptive model-selected roles.Source: §I · p. 1
  • 12–44% higher accuracy than zero-shot and zero-shot-CoT baselines was reported across 20 LLMs and seven benchmarks, with the largest gains in arithmetic tasks.Domain-specific roles such as Mathematician outperformed general roles on structured arithmetic tasks.Source: §I · p. 1
  • The paper identifies task-dependent role specialization and prompt-engineering insights, including pre-question prompts and adaptive selection for open-weight models.Its ablations examine nontraditional roles and lexical variations.Source: §I · p. 1

A. Chain-of-Thought Prompting for Reasoning.

  • CoT prompting elicits intermediate reasoning steps, but zero-shot prompts remain phrasing-sensitive; native reasoning models add inference-time latency and overhead.RE-ZS-CoT explores domain-specific personas as an alternative direction for standard, non-RL-tuned models.Source: A. Chain-of-Thought Prompting for… · p. 2

B. Role-Based Prompting

Prior role-prompting work was limited in scale and flexibility, motivating an evaluation of role choice and placement across tasks and models. The section describes the framework, benchmark setup, role-selection procedures, and baseline comparisons.

B. Role-Based Prompting

  • Prior role-prompting studies relied on curated examples, narrow domains, or manual templates, while large-scale zero-shot-CoT evaluations remained limited.Earlier math role-play prompts sometimes failed or degraded performance, motivating broader evaluation of role selection and placement.Source: B. Role-Based Prompting · p. 2

III. ROLE-ENHANCED ZERO-SHOT CHAIN OF THOUGHT

  • RE-ZS-CoT tests eight predefined expert roles and a dynamic version in which each model selects one role for each benchmark.It uses a single prompt to extract answers rather than the two-step zero-shot-CoT procedure described in the passage.Source: §III · p. 3
  • The study isolates role-selection and role-placement effects against zero-shot and zero-shot-CoT baselines, not other role-prompting frameworks.The authors identify these mechanics as underexplored dimensions.Source: §III · p. 3

A. Prompt Variants

  • The prompt combines a role R and question Q in three placements: R→Q, Q→R, and R→Q→R.These variants test whether role-phrase placement affects performance.Source: A. Prompt Variants · p. 3, Figure 2 · p. 3

IV. IMPLEMENTATION DETAILS

  • The evaluation spans arithmetic, commonsense, and reading comprehension datasets, including AQuA-RAT, GSM8K, OpenBookQA, StrategyQA, RACE, and TriviaQA.The experiments use 20 models, ranging from 2b–14b open-weight models to larger OpenAI models.Source: §IV · p. 3
  • Zero-shot and zero-shot-CoT share a prompt except for “Let’s think step by step”; all prompts use a one-stage comparison setup.Answer detection takes the earliest compliant answer token, normalizes it, and compares it with the answer label.Source: §IV · p. 4
  • For open-weight models, static self-alignment selects the two most frequently chosen roles after 100 model queries per task.The procedure uses lightweight meta-prompts rather than testing every role-prompt combination, in part to accommodate computational constraints.Source: §IV · p. 4
  • Dynamic self-alignment queries each model 100 times without a predefined role list, then uses its most frequently generated role.The model infers a task-specific role from the benchmark description.Source: §IV · p. 4

A. Overall Results

RE-ZS-CoT outperforms zero-shot and zero-shot-CoT across most benchmarks, especially arithmetic and reading comprehension. Static and dynamic results generally align, while role and prompt-position performance varies by task and model.

A. Overall Results

  • RE-ZS-CoT outperforms zero-shot and zero-shot-CoT across most benchmarks, with especially pronounced gains in arithmetic and reading comprehension.Tables I and II compare the two RE-ZS-CoT versions with both baselines across the listed benchmarks.Source: Table I · p. 5, Table II · p. 5, A. Overall Results · p. 5
  • Benchmarks improved by static roles almost always also improve with dynamic roles; static often peaks on arithmetic, while dynamic often leads on reading comprehension and commonsense tasks.Across all model-task pairs, RE-ZS-CoT beats the stronger baseline in approximately two-thirds of instances.Source: A. Overall Results · pp. 5–6

B. Nonconventional Roles

  • Unconventional roles diminish performance across all tested LLMs, with only two exceptions.The experiment tested Poet, Comedian, and Politician roles.Source: B. Nonconventional Roles · p. 6

C. Role-Phrase Position and Static Self-Alignment Details

  • No prompt-position variant is consistently superior; the best placement depends on both the LLM and benchmark.Pre-question leads on AQuA-RAT, while split leads on GSM8K.Source: C. Role-Phrase Position and Static… · p. 6
  • Mathematician, Literacy Specialist, and Polymath appear to yield the best performance overall.Mathematicians are among the roles associated with superior performance on arithmetic reasoning.Source: C. Role-Phrase Position and Static… · p. 6

VI. ABLATIONS

The discussion finds that role effectiveness depends on task alignment and prompt placement: static selection can peak on specialized tasks, while dynamic selection often helps on broader ones. It also identifies limits in error analysis, statistical testing, model and language scope, and training-data transparency.

VII. DISCUSSION

  • Unconventional roles lower performance, and replacing or removing “expert” generally reduces accuracy; “top” improves two AQuA-RAT cases.The discussion attributes the pattern to task-role alignment and explicit expertise cues.Source: §VI · p. 6, §VII · p. 7
  • Static role selection peaks on specialized tasks such as AQuA-RAT, while dynamic selection often improves commonsense and reading-comprehension outcomes.The discussion describes dynamic selection as more robust for less prescribed reasoning paths, though it does not always reach static selection’s ceiling on specialized tasks.Source: §VII · p. 7
  • RE-ZS-CoT is presented as a lightweight alternative to costly, token-heavy native reasoning, with results on open-weight models spanning 2b to 14b parameters.The discussion frames this as relevant to resource-constrained settings where heavy-compute System 2 models are unfeasible.Source: §VII · pp. 7–8
  • Opaque training data, especially for closed-source models, makes reproducibility difficult and limits fairness and bias evaluation of how models define expert roles.The authors note that training-data composition influences how a model defines and operationalizes an expert role.Source: §VII · p. 8
  • The evaluation does not include deep qualitative or mechanistic error analysis, leaving the causes of degraded OpenBookQA performance unresolved.The authors hypothesize rigid role constraints may induce overthinking in intuitive contexts, but call for interpretability research.Source: §VII · p. 8
  • Absolute accuracy gains are reported without paired significance tests for every model-task pair, so small single-digit gains on limited datasets are competitive rather than definitive.The authors contrast these cases with macro-trends and double-digit surges such as those on AQuA-RAT.Source: §VII · p. 8
  • The study focuses on standard autoregressive models and English text-only corpora, leaving RL-driven reasoning, non-English, conversational, and multimodal settings unexplored.Future work also proposes testing two-stage prompting, few-shot methods, and multi-role collaboration.Source: §VII · pp. 8–9

VIII. CONCLUSION

RE-ZS-CoT combines domain-specific roles with chain-of-thought prompting to improve zero-shot reasoning across seven benchmarks and 20 models. The conclusion reports accuracy gains of up to 44 percentage points and highlights role alignment and low-resource scalability.

VIII. CONCLUSION

  • RE-ZS-CoT improved zero-shot reasoning across seven benchmarks and 20 models, with accuracy gains of up to 44 percentage points.The method integrates domain-specific roles with chain-of-thought prompting without fine-tuning or multi-turn interactions.Source: §VIII · p. 9
  • The findings identify domain alignment and role-phrase position as important considerations in prompt design.The conclusion also characterizes the framework as efficient and scalable for low-resource settings.Source: §VIII · p. 9
  • The authors present role-aware prompting as a bridge between generic prompting and domain-specific expertise.The paper frames this as a way to advance zero-shot reasoning without relying on fine-tuning or multi-turn interactions.Source: §VIII · p. 9

B. Datasets

The section describes the datasets used for evaluation and identifies the test splits, sample counts, and task categories reported in Table VI.

  • Table VI reports each dataset’s test split and sample count, and notes that TriviaQA was downsampled to 1,000 records to save computational resources.Source: Table VI · p. 10
  • AQuA-RAT contains algebraic word problems with step-by-step natural-language solutions, while GSM8K tests multi-step arithmetic reasoning across problems requiring two to eight steps.Source: B. Datasets · p. 10
  • OpenBookQA questions involve multi-step reasoning, commonsense knowledge, and text interpretation; StrategyQA questions require strategic inference and include decompositions and supporting evidence.Source: B. Datasets · p. 10
  • RACE comprises reading-comprehension passages and questions from middle- and high-school English examinations in China, while TriviaQA pairs trivia questions with independently gathered evidence documents.Source: B. Datasets · p. 10

C. Models and Implementation Details

The study evaluates 20 models, describes the implementation setup, and reports a MATH benchmark comparison in which the selected role was consistent across models.

  • The experiments used 20 LLMs, including open-weight models from 2b to 14b and closed-source OpenAI models.The full model list includes GPT-4.1, GPT-4o, and open-weight model families such as Phi4, Qwen2.5, Gemma, and Granite.Source: C. Models and Implementation Details · pp. 10–11
  • Models were queried once per dataset record at temperature zero with a fixed random seed, using Ollama for local open-weight execution and the OpenAI API.The experiments used an NVIDIA A100 80GB system and a two-GPU NVIDIA RTX 6000 Ada system, totaling approximately 5,600 GPU hours.Source: C. Models and Implementation Details · p. 11
  • The MATH evaluation used 5,000 test questions across seven mathematical categories and added output-format instructions to support evaluation.Source: C. Models and Implementation Details · p. 11
  • RE-ZS-CoT consistently outperformed the two baselines on MATH, and “Mathematician” was the top-selected role across all LLMs among the eight roles tested.Table VII presents results across seven MATH topics and compares the optimal role and variant for each LLM with zero-shot and zero-shot-CoT.Source: C. Models and Implementation Details · p. 11

D. Static Self-Alignment

The section describes model-driven role selection and reports stronger self-alignment for selected GPT models than for open-weight models.

  • For three smaller GPT variants, models selected roles from a predefined list 100 times, enabling assessment of whether their two best-performing roles were among their most frequent selections.Tables VIII–XIII present static self-alignment results for the GPT models.Source: D. Static Self-Alignment · p. 11
  • Selected GPT models showed stronger self-alignment than open-weight models, indicating better self-alignment capability among larger models.The authors identify improved self-alignment in smaller models as a need.Source: D. Static Self-Alignment · p. 11
  • Open-weight models were each queried 100 times to select roles from a predefined list, and the two most frequently selected roles were used in experiments.Tables XIV–XIX report the self-alignment results.Source: D. Static Self-Alignment · p. 11

E. Dynamic Self-Alignment

The section provides tables of static and dynamically selected role results across models and benchmarks, plus a figure on role effects and self-alignment.

Related papers

Loading 2610.29073v1…