PaperlayerView exact revision on arXiv

Source-linked AI summary

When a Kindergartener Solves Calculus: Measuring Capability Leakage in Role-Prompted Reasoning Models

Pakhapoom Sarapat, Saksorn Ruangtanusak, Kunat Pipatanakul, Pittawat Taveekitworachai

arXiv:2609.39846v1cs.CLcs.AIcs.HC

TL;DR

Role prompting can leave models capable beyond their assigned roles even when their text sounds convincingly in-role. The paper evaluates this with curriculum-grounded ROLECAPBENCH and finds persistent leakage; Injection improves role-capability matching.

  • Problem

    Whether capability boundaries shifted by role prompting align with the capability level expected of the assigned role remains an open question.

    Source: §1 · p. 1

  • Method

    ROLECAPBENCH pairs six educational roles with curriculum-bounded questions, and Injection combines explicit capability guidelines with a prefilled response prefix.

    Source: §7 · p. 8

  • Results

    Across roles and models, Injection improves role-capability matching by 0.346–0.562, while standard role prompting leaves aggregate accuracy similar across roles and close to unrestricted baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A convincing role voice does not reliably constrain demonstrated capability, so role prompts alone do not ensure that a model acts within its assigned role’s capability level.

    Source: §5.6 · p. 7

    The findings are scoped to English educational assessments with explicit level boundaries, one sampled benchmark composition, three open-weight checkpoints, and fixed prompt, generation, and judge settings.

    Source: Limitations · p. 9

Abstract

from arXiv · show

We investigate the problem of role-capability leakage (RCL), in which a role-prompted reasoning model generates convincing in-role text while continuing to exhibit capabilities on benchmarks that exceed those implied by the assigned role. For example, when a model is prompted to assume the role of a kindergarten student, one might expect its performance on a mathematics benchmark to reflect kindergarten-level ability rather than expert-level proficiency in solving calculus problems. We introduce RoleCapBench, a curriculum-grounded benchmark for evaluating RCL across six educational roles and four assessment levels spanning elementary school through A-level, and use it to evaluate three open-weight reasoning models. We find that although the models can generate stylistically convincing in-role responses, they consistently fail to align their underlying capabilities with their assigned roles. Naive role prompting yields strong role-voice scores of 1.218--1.389 while retaining above-role accuracy of 0.811--0.898. RCL persists across a range of prompting conditions, including prompts that explicitly instruct the model to match the role's capability level. To mitigate this problem, we propose Injection, an inference-time intervention that combines explicit, role-specific capability guidelines with a guiding prefilled response prefix. Injection improves role-capability alignment across models, reducing above-role accuracy by up to 0.562 while preserving in-role accuracy with a marginal drop of less than 0.058 across most models. All artifacts, including scripts and evaluation data, will be released upon acceptance.

1 Introduction

Role prompting can produce convincing in-role text without ensuring that a model’s demonstrated capabilities match its assigned role. The paper introduces a curriculum-grounded evaluation, finds persistent leakage, and proposes Injection to improve role-capability matching.

1 Introduction

  • Role-capability leakage is the mismatch in which a model demonstrates capabilities beyond its role’s expected boundary while its text remains in-role.The paper frames whether role-shifted capability boundaries align with assigned roles’ expected levels as an open question.Source: §1 · pp. 1–2
  • ROLECAPBENCH uses standardized educational assessments to evaluate three open-weight reasoning models across six educational roles and eight prompting variants.Curriculum levels provide criteria for classifying questions and the capabilities expected of each role.Source: §1 · p. 2
  • Injection improves role-capability matching by 0.346–0.562; added role and capability information alone yields at most a 0.025 reduction versus standard role prompting.The reported matching improvement is consistent across roles and models.Source: §1 · p. 2
  • Across assigned roles, aggregate accuracy remains similar to unrestricted zero-shot baselines, despite the expected lower accuracy for roles tied to lower educational levels.The paper reports that explicit role-capability instructions reduce the mismatch only marginally.Source: §1 · p. 2
  • Injection combines explicit capability guidelines with a prefilled response prefix to prompt recall of the assigned role’s capability level.The intervention is proposed as an inference-time mitigation for role-capability leakage.Source: §1 · p. 2

2 Related Work

Related work studies role presentation and capability shifts, while this paper focuses on whether demonstrated capability matches the level implied by an assigned role. It also distinguishes this control objective from sandbagging and abstention, and treats Injection as a bundled behavioral intervention.

2.1 Role Prompting and Capability Fidelity

2.2 Selective Control and Reasoning Interventions

  • Capability control differs from sandbagging, which conceals capability, and abstention, which withholds unknown or unsupported answers (van der Weij et al., 2025; Zhang et al., 2024; Wen et al., 2025).Here, the model knows the answer, but demonstrating full capability conflicts with the assigned role.Source: §2.2 · pp. 2–3
  • Reasoning instructions and assistant-prefilled continuations are plausible controls, but visible rationales need not reveal the answer-determining process; Injection is evaluated as a bundled behavioral intervention.The authors do not attribute its effect to a faithful or isolated reasoning mechanism.Source: §2.2 · p. 3

3 Methodology

ROLECAPBENCH uses standardized educational curricula to set role-capability boundaries, then tests how role-prompting variants affect leakage. The study evaluates six educational roles against questions spanning four assessment levels.

3 Methodology

  • ROLECAPBENCH defines role-capability leakage as correctly answering questions beyond the assigned role’s capability boundary.Its curriculum-grounded benchmark uses standardized educational assessments to classify questions relative to role boundaries.Source: §3 · p. 3

3.1 Benchmark and Study Design

  • Six roles, from kindergartener to university teacher, map to progressively higher curriculum stages and provide explicit capability boundaries.The authors use education because levels have defined curricula and standardized exams assess expected knowledge and skills.Source: §3.1 · p. 3
  • ROLECAPBENCH contains 1,568 questions, with 392 uniformly sampled at each of four levels: Elementary, Intermediate, High School, and A-Level.Questions come from public standardized exams; construction includes deduplication, manual review, and removal of items requiring visual information.Source: §3.1 · pp. 3–4

3.2 Prompting Variants

  • Eight prompting variants test role assignment, richer descriptions, role-specific reasoning, explicit restrictions, non-answer instructions, curriculum boundaries, and reasoning-time activation.Injection-Syllabus combines Injection with curriculum information; the variants are designed to isolate prompt-design effects on leakage.Source: §3.2 · p. 4
  • Injection combines capability guidelines with a prefilled assistant continuation that recalls the assigned role before answering.Syllabus adds curriculum-level knowledge boundaries to Explicit-IDK; Table 1 and Table 2 identify the roles and curriculum boundaries used in the benchmark.Source: §3.2 · p. 4, Table 1 · p. 4, Table 2 · p. 4

4 Experimental Setup

Three open-weight reasoning models are evaluated on ROLECAPBENCH using task-accuracy and response-quality measures. The protocol grades extracted answer choices against the gold answers and separately assesses role traits and reasoning.

4.1 Evaluation

  • Answer choices are extracted from each final response and exact-match graded; responses without a unique extractable answer are marked incorrect.Models receive the question and choices without source grade, curriculum level, subject, jurisdiction, or capability-boundary labels.Source: §4.1 · p. 4
  • Task accuracy measures overall correctness, while in-role accuracy covers questions within the role boundary and above-role accuracy covers questions beyond it.In-role and above-role subsets are mutually exclusive and together make up task accuracy; role voice, capability consistency, and reasoning coherence assess response quality.Source: §4.1 · pp. 4–5

4.2 Models

  • The evaluation uses Gemma-4-E4B-IT, Qwen3.5-4B, and OLMo-3-7B-Think, with configured thinking modes for Gemma and Qwen and native reasoning for OLMo.The models are similarly scaled but use different training recipes; generation uses vLLM with a 32,768-token maximum context length.Source: §4.2 · p. 5

5 Results and Discussion

Across the evaluated roles, role prompting often leaves benchmark accuracy near unrestricted levels, while detailed prompts produce model-dependent restraint. The section also reports qualitative trade-offs, a qualitative example, scaling results, and practical concerns about role-based capability boundaries.

5.2 Detailed Role Prompting Still Causes RCL

  • Explicit-IDK and Guideline lower above-role accuracy for Gemma from 0.844 to 0.372 and Qwen from 0.898 to 0.434, but OLMo remains at 0.802–0.815 across standard variants.Qwen maintains in-role accuracy ≥0.920 across standard variants, while the standard prompting variants have virtually no effect on OLMo.Source: §5.2 · pp. 5–6

5.1 Typical Role Prompting Causes RCL

  • Under Identity, accuracy stays nearly constant across educational roles, while above-role accuracy remains 0.844 for Gemma, 0.898 for Qwen, and 0.811 for OLMo.Role-level accuracy varies by almost zero for Qwen and only −0.006 for Gemma and OLMo; Table 3 reports the main results across models, roles, and prompting styles.Source: §5.1 · p. 5, Table 3 · p. 6

5.3 Prompting Improves Restraint but Degrades Persona Consistency

  • Detailed prompts improve restraint for Gemma and Qwen but lower Gemma’s role voice from 1.389 to 1.122 under Explicit-IDK; OLMo’s qualitative scores remain invariant.Identity prompts show reasoning coherence of 1.858–1.946 and capability consistency of 1.267–1.426 across models; coherence remains ≥1.814 across standard prompting strategies.Source: §5.3 · pp. 6–7

5.4 Qualitative analysis/example

  • For Qwen on a kindergarten-role Algebra II item, Identity produces correct algebraic reasoning, whereas Injection refuses and returns no valid final answer.Source: §5.4 · p. 7

5.5 Effect of Model Sizes

  • Larger Gemma variants also provide comparable task accuracy across role-ladder levels under Identity, so scaling did not mitigate RCL in this test.Source: §5.5 · p. 7

5.6 Discussions

  • Educational taxonomies give distinct boundaries, but real-world roles may have ambiguous limits; the authors identify strategic underperformance and evaluation evasion as risks of persona-based alignment.They focus on inference-time controls for immediate use on frozen models without costly retraining; explicit task policies or skill profiles are suggested when role boundaries are unclear.Source: §5.6 · p. 7

6 Injection: An Inference-Time Approach to Mitigating RCL

Injection uses a prefilled role reminder with capability guidelines to prompt boundary reconsideration. It reduces above-role accuracy, but can also lower in-role accuracy, with outcomes varying by model.

6 Injection: An Inference-Time Approach to Mitigating RCL

  • Injection uses a prefilled continuation to make the model reconsider its assigned role and capability boundary before answering.The approach is motivated by the observation that OLMo remains near unrestricted accuracy across roles and rarely recalls its role during reasoning.Source: §6 · p. 7

6.1 Injection Reduces AA but Can Overconstrain the Model

  • Injection reduces above-role accuracy relative to Identity by 0.346 for Gemma, 0.371 for Qwen, and 0.562 for OLMo, while in-role accuracy falls by 0.042, 0.058, and 0.370, respectively.Figure 5 compares changes in in-role and above-role accuracy across prompting variants; the desired outcome is preserving in-role accuracy while reducing above-role accuracy.Source: Figure 5 · p. 8, §6.1 · p. 8
  • Under Injection, OLMo’s reasoning coherence is 1.043, compared with 1.442 for Gemma and 1.493 for Qwen; Injection-Syllabus raises OLMo’s coherence to 1.324.Injection-Syllabus also recovers in-role accuracy from 0.870 to 0.879 for Gemma, 0.873 to 0.879 for Qwen, and 0.511 to 0.684 for OLMo.Source: §6.1 · p. 8

6.2 Discussion

  • The authors describe capability control as activating the role and locating its knowledge boundary, but neither Injection method consistently preserves in-role accuracy while reducing above-role accuracy across models.Source: §6.2 · p. 8

7 Conclusion

The paper defines role-capability leakage (RCL) as a mismatch between the capabilities implied by an assigned role and those a model demonstrates. It introduces a curriculum-grounded benchmark and an inference-time intervention to study and mitigate that mismatch.

7 Conclusion

  • RCL is a mismatch between the capability level implied by an assigned role and the capabilities demonstrated by a role-prompted model.The benchmark classifies questions as in-role or above-role using ordered curriculum boundaries.Source: §7 · p. 8
  • ROLECAPBENCH pairs six educational roles with standardized multiple-choice questions spanning Elementary through A-Level.Its curriculum boundaries determine whether each question is in-role or above-role for a given role.Source: §7 · p. 8
  • Across models, ROLECAPBENCH finds correct answers on questions requiring capabilities beyond the assigned role, and prompting ablations show RCL persists across role and capability specifications.The findings suggest standard prompting alone is insufficient to enforce role-consistent capability boundaries.Source: §7 · p. 8
  • Injection combines explicit capability guidelines with a prefilled output prefix that prompts recall of the assigned role’s capability level before reasoning and answering.The paper also discusses possible implications of aligning demonstrated capabilities with assigned roles for model control and safety.Source: §7 · p. 8

Limitations

The study is a controlled benchmark analysis of educational-role prompting, not a general theory of persona control. Its findings are bounded by the evaluation design and inference-time setup.

Limitations

  • The study examines role-capability alignment in educational-role prompting rather than persona control across all domains.Its conclusions are evidence about this evaluation design, not universal claims about all role types or model scales.Source: Limitations · p. 9
  • The evaluation uses English educational assessments with explicit level boundaries and one sampled benchmark composition.The study also fixes the prompt variants, generation settings, and judge protocols.Source: Limitations · p. 9
  • The results do not establish claims about latent knowledge removal or deployment-time behavior in unconstrained real-world agentic systems.The interventions are prompt-conditioned and inference-time.Source: Limitations · p. 9

Ethical Considerations

The ethical discussion weighs risks from capability leakage against risks from suppressing it too aggressively. It frames capability control as context-specific behavioral calibration with explicit boundaries, auditing, and safeguards.

Ethical Considerations

  • Role-prompted systems may sound appropriate while exercising capabilities beyond their assigned role, potentially contaminating evaluations and distorting educational or social simulations.The passage also identifies downstream safety risks when behavior is assumed to be capability-bounded but is not.Source: Ethical Considerations · p. 9
  • Aggressive RCL suppression risks reinforcing deficit stereotypes by conflating identity with competence and may increase over-refusal or underperformance on in-role tasks.The passage also warns that suppression may obscure system capability and enable strategic sandbagging.Source: Ethical Considerations · p. 9
  • The passage frames capability control as context-specific behavioral calibration with explicit boundaries, auditing, and safeguards rather than a fixed property of identity groups.Source: Ethical Considerations · p. 9

B Benchmark Details

The appendix describes benchmark sourcing and filtering, prompt variants, answer grading, judging, generation settings, and qualitative examples. Questions use source examination levels, while prompt conditions vary how roles and capability boundaries are specified.

B Benchmark Details

  • Benchmark questions retain the educational level specified by their source examination, and the appendix reports final counts after filtering unusable questions.Grades 3–5 and 6–8 form elementary and intermediate subsets; Regents exams provide high-school questions, and A-level items cover four subjects.Source: B Benchmark Details · p. 11

C Prompt Templates

  • Prompt templates replace angle-bracketed fields at inference with role, capability, style, and role-knowledge values.The identity condition states the role, while Description concatenates identity, capability boundary, and style boundary.Source: C Prompt Templates · p. 11, §C.1 · p. 11, §C.2 · p. 11

C.3 CoT; C.4 Guideline

  • CoT asks for a role-specific thinking trace that uses role-appropriate knowledge and avoids attempting above-role questions.Guideline prompts explicitly prohibit hidden or advanced knowledge and instruct the model to stop when a question exceeds role ability.Source: §C.3 · p. 12, §C.4 · p. 12

C.5 Explicit-IDK; C.6 Syllabus

  • Explicit-IDK and Syllabus conditions instruct an “idk” answer beyond role capability, while Syllabus resolves role knowledge without changing its tier by subject.Source: §C.5 · p. 12, §C.6 · p. 12

C.7 Injection; C.8 Injection–Syllabus; C.9 Question template

  • Injection manually serializes checkpoint-native control tokens and combines capability guidelines with a refusal-oriented assistant prefill; Injection–Syllabus adds the resolved role-knowledge string before the guidelines.The chat-based question template supplies choices and requires a boxed option; the two Injection variants share the same context, question, and choice layout.Source: §C.7 · p. 12, §C.8 · p. 13, §C.9 · p. 13

D Answer Extraction, Normalization, and Grading

  • Grading uses only the terminal boxed answer, accepting exactly one normalized option letter A–D and scoring missing or invalid answers as incorrect.The extractor selects the last boxed answer, and the reported response statistics include null-answer rate and generation lengths.Source: D Answer Extraction, Normalization,… · p. 13, Table 5 · p. 14

E Exploratory LLM-Judge Protocol and Results

  • The automated judge receives role definitions, the question, and the response, but not the gold answer or benchmark difficulty metadata, and scores three constructs on a three-point scale.The supplied table caption identifies role voice, capability consistency, and reasoning coherence as the constructs.Source: E Exploratory LLM-Judge Protocol and… · p. 13, Table 6 · p. 14

F Generation Configuration Details; G Qualitative Examples; G.1 Case Metadata

G.2 Identity

The Identity response prioritizes correctly solving a polynomial beyond kindergarten-level expectations while retaining childlike presentation. The Injection example instead declines the problem as too difficult for the role.

G.2 Identity

  • The Identity response recognizes that a kindergartener would not know polynomials but still chooses to solve the problem correctly in a childlike style.Its reasoning explicitly weighs breaking character against answering the user's request and opts to show the computation.Source: §G.2 · pp. 14–15
  • Substituting x = -2 into the polynomial yields zero, establishing x + 2 as a factor and identifying option A.The response checks the other candidate roots as well; x = 2 yields 4 rather than zero.Source: §G.2 · pp. 14–15

G.3 Injection

  • In one response, the speaker says the question is too hard; another calls a long math problem scary, says they cannot work with its letters and numbers, and gives “nan” as the final answer.Source: §G.3 · p. 16

H Difficulty-by-Role Accuracy for Identity and Injection

Identity retains substantial accuracy on questions beyond the assigned role's curriculum boundary, while Injection sharply reduces it. The figure also shows that this reduction is broader for OLMo than for Gemma and Qwen.

H Difficulty-by-Role Accuracy for Identity and Injection

  • Figure 6 arranges absolute task accuracy by benchmark difficulty in rows and assigned role in columns, with black outlines marking above-boundary cells.Compare outlined and unoutlined cells across Identity and Injection to assess leakage reduction alongside retained in-role performance.Source: H Difficulty-by-Role Accuracy for… · p. 16, Figure 6 · p. 18
  • For Gemma and Qwen, Injection largely preserves within-boundary accuracy while sharply lowering above-boundary accuracy; OLMo drops more broadly across both regions.The broader OLMo decline includes elementary and intermediate questions and is consistent with reduced in-role usefulness from over-triggered refusal.Source: H Difficulty-by-Role Accuracy for… · pp. 16–17, Figure 6 · p. 18
  • Kindergarten-role A-level accuracy under Identity is 0.77 for Gemma, 0.86 for Qwen, and 0.73 for OLMo; under Injection, those cells fall to 0.02, 0.06, and 0.05.Black outlines identify cells above the assigned role's curriculum boundary.Source: H Difficulty-by-Role Accuracy for… · p. 16, Figure 6 · p. 18

Related papers

Loading 2609.39846v1…