PaperlayerView exact revision on arXiv

Source-linked AI summary

R$^2$A: Learning Persona Policies Through Persona Representation Learning and Runtime Alignment

Mohan Zhang, Chengsong You, Xiaoyu Cao, Zhen Sun, Xiaohan Jia, Junwei Zhou, Yongchao Chen

arXiv:2608.29798v1cs.CLcs.AI

TL;DR

Static Persona elicitation can help in one task but hurt in another, motivating a framework that separates Persona state selection from trajectory realization. R2A learns structured Persona representations and then aligns both components with task feedback, outperforming the base model and static elicitation across 12 settings.

  • Problem

    Static Persona elicitation produces inconsistent task effects because the same high-level Persona specification does not reliably translate into context-appropriate behavior.

  • Method

    R2A first learns the target Persona through structured Who–How–What presentations, then performs prompt-free Runtime Alignment to calibrate selection and realization using task feedback.

  • Results

    Across 12 evaluation settings, R2A achieves more consistent performance than both the base model and static Persona elicitation.

  • Takeaways & Limitations

    Persona policy learning benefits from representing conditional behavioral principles and trajectory-level manifestations before runtime alignment.

  • Takeaways & Limitations

    R2A remains below the base model on AbstentionBench-fast and OR-Bench-Hard, with remaining regressions concentrated in Feasibility Honesty.

Abstract

from arXiv · show

The same Persona behavior can be beneficial in one context but harmful in another, causing static Persona elicitation to perform inconsistently across tasks. We introduce the Persona Selection--Realization Framework, which models behavior generation through a latent Persona state and decomposes it into Persona Selection and Persona Realization. The discrepancies between static Persona elicitation and an ideal Persona policy in these two components define the Selection Gap and Realization Gap, respectively. Building on this framework, we propose R$^2$A, a two-stage approach for learning Persona policies. Persona Representation Learning uses structured Who--How--What presentations to encode the target Persona's objective, conditional behavioral principles, and trajectory-level manifestations. Persona Runtime Alignment then removes the explicit Persona specification and jointly calibrates behavior selection and trajectory realization using task feedback. Across 12 evaluation settings covering the four principles of the Accountable-Professional Persona studied in this work, R$^2$A overall outperforms both the base model and static Persona elicitation. Ablation results further show that Persona Representation Learning is critical for preventing Runtime Alignment from producing behaviorally imbalanced policies and for achieving more stable Persona policy learning.

1 Introduction

Static Persona elicitation can change model behavior, but its effects vary across tasks. The paper introduces a framework and R2A to explain and reduce this inconsistency.

  • Persona specifications steer LLM behavior by describing identity, objectives, and behavioral principles.
  • The Accountable-Professional Persona evaluates four principles through paired signature and complementary behaviors with corresponding benchmark probes.
  • Static elicitation improves, barely changes, or degrades performance across tasks despite using the same Persona specification.
  • The paper frames the central question as how static elicitation translates a high-level Persona into concrete task behavior and why performance varies across tasks.
  • The Persona Selection–Realization Framework models task behavior through a latent Persona state and separates selecting that state from realizing it in a complete output trajectory.
  • R2A learns Persona representations through structured Who–How–What presentations, then calibrates behavior selection and trajectory realization using task feedback.

2 Persona Selection–Realization Framework

The framework represents task behavior as selecting a latent Persona state and realizing it as a complete trajectory. It explains static elicitation’s task-dependent effects through Selection and Realization deviations from an ideal policy.

  • 2.1 Accountable-Professional Persona and Static Elicitation: The Accountable-Professional Persona comprises four principles: Information Sufficiency, Feasibility Honesty, Premise Scrutiny, and Evidence Responsiveness.
  • 2.1 Accountable-Professional Persona and Static Elicitation: Static elicitation differs from the base condition only by adding a fixed Persona specification to the inference input.
  • 2.1 Accountable-Professional Persona and Static Elicitation: The same Persona prompt improves some tasks, has limited effects on others, and degrades performance in several cases.
  • 2.2 Persona Selection and Persona Realization: A continuous latent state z represents the task-conditioned Persona instantiation, which organizes subsequent decisions and complete output trajectories.
  • 2.2 Persona Selection and Persona Realization: Persona Selection determines the applicable state from task context and specification, while Persona Realization turns that state into concrete behavior and a complete trajectory.
  • 2.2 Persona Selection and Persona Realization: The decomposition treats z as the mediator through which the external specification affects trajectory generation, without requiring separate implementation modules.
  • 2.3 How Static Persona Elicitation Acts: The static specification can induce task-dependent selection shifts, while each Persona state’s expected utility also varies across tasks.
  • 2.3 How Static Persona Elicitation Acts: The resulting value change may be positive, near zero, or negative, explaining inconsistent static-elicitation effects across tasks.

3 R2A: Representation-to-Alignment

R2A learns an adaptive Persona policy in two stages: structured representation learning followed by runtime alignment without an explicit Persona specification. It jointly calibrates context-appropriate behavior selection and the quality of its complete trajectory.

  • 3.2 Persona Runtime Alignment: Runtime Alignment removes the explicit Persona specification and trains the first-stage policy under the default condition to infer actions from task context.Its data cover complementary task states associated with each behavioral principle.
  • 3.1 Persona Representation Learning: R2A first teaches the target Persona through Who, How, and What presentations covering objectives, conditional principles, and complete task trajectories.Who defines identity and responsibility, How specifies applicability boundaries, and What grounds the principles in concrete task performance.
  • 3.1 Persona Representation Learning: The first-stage representation policy is trained on three supervised datasets with a weighted language-modeling objective.The weights λd control the contribution of each presentation type.
  • 3.2 Persona Runtime Alignment: A high-level behavior extractor makes Persona Selection observable by mapping each complete trajectory to a course of action such as ASK, ANSWER, or CRITIQUE.The extracted behavior is an observable proxy for the latent Persona state rather than the state itself.
  • 3.2 Persona Runtime Alignment: Runtime Alignment assigns target behaviors from task conditions and combines behavior correctness with trajectory quality into task rewards.The behavior reward checks whether the required action is selected, while the trajectory reward evaluates how well it is realized.
  • 3.2 Persona Runtime Alignment: Group Relative Policy Optimization updates the aligned policy using relative candidate rewards while a KL penalty constrains deviation from the first-stage policy.The coefficient β controls deviation from the first-stage policy.

4 Experiments

Experiments evaluate R2A across four Accountable-Professional principles and 12 settings, comparing it with Qwen3-8B, static Persona elicitation, and RA-only. R2A delivers more consistent gains, preserves complementary behaviors, and improves the balance between intervention and non-intervention behavior.

  • Experimental setup: Experiments compare Qwen3-8B, AP Persona, RA-only, and R2A across four Accountable-Professional principles using benchmark-specific primary metrics.The evaluation covers Information Sufficiency, Feasibility Honesty, Premise Scrutiny, and Evidence Responsiveness.
  • Main results: AP Persona improves over Qwen3-8B in 6 of 12 settings but degrades performance in the other 6, whereas R2A outperforms Qwen3-8B in 10 settings.Relative comparisons use task-level change directions because benchmark metrics have different scales.
  • Main results: R2A improves ClarifyMT-Clarify from 70.96 to 85.10 and ClarifyMT-Answer from 72.99 to 77.25, avoiding AP Persona’s pronounced over-clarification.R2A also achieves the best results on both QuestBench subsets and reverses AP Persona regressions on both Evidence Responsiveness benchmarks.
  • Persona Representation Learning: R2A outperforms RA-only in 10 of 12 settings, while RA-only shows extreme ANSWER bias on ClarifyMT with 99.90% Answer and 0.00% Clarify.The ablation differs only in whether Persona Representation Learning precedes Runtime Alignment.
  • Behavioral Frontier Analysis: The behavioral frontier measures signature-side and complementary-side success across 672 questions spanning Information Sufficiency, Feasibility Honesty, and Premise Scrutiny.Both axes assign equal weight to the three principles, and higher values are better.
  • Behavioral Frontier Analysis: R2A lies at (75.33, 82.11), a signature/complementary behavioral combination reached by none of the evaluated static Persona prompts.Relative to AP Persona, R2A improves signature-side and complementary-side success by 6.11 and 11.48 percentage points.

5 Conclusion

The paper introduces the Persona Selection–Realization Framework and R2A, which learns Persona policies without explicit Persona prompts at inference time. Across 12 settings, R2A is more consistent than both the base model and static Persona elicitation, while representation learning helps balance complementary behaviors.

  • Conclusion: The Persona Selection–Realization Framework models latent Persona states, separates Persona Selection from Persona Realization, and defines Selection and Realization Gaps.These gaps formalize deviations between static elicitation and an ideal Persona policy.
  • Conclusion: R2A combines Persona Representation Learning with Runtime Alignment to learn Persona policies without explicit Persona prompts at inference time.The two-stage approach is built on the proposed framework.
  • Conclusion: Across 12 evaluation settings, R2A achieves more consistent performance than both the base model and static Persona elicitation.Ablation and behavioral-frontier analyses further support the importance of representation learning for balancing signature and complementary behaviors.

A.1 Full Persona Specification

The experiments use a verbatim Accountable-Professional specification and test whether the same two-stage procedure extends to Qwen3-14B. On the larger backbone, R2A retains an advantage over both the base model and static elicitation.

  • Full Persona Specification: The experiments use a verbatim Accountable-Professional specification organized around four commitments for task handling.The supplied specification begins by requiring clarification when information is missing and direct action otherwise, alongside further commitments.
  • Full Persona Specification: The same two-stage training procedure is applied to Qwen3-14B and evaluated on the same 12 settings against the original and statically elicited Qwen3-14B.This tests whether R2A scales to a larger backbone.
  • Full Persona Specification: On Qwen3-14B, R2A records 8 wins and 4 losses versus the base model and 9 wins and 3 losses versus static Persona elicitation.R2A achieves the best result on every Premise Scrutiny and Evidence Responsiveness evaluation and three of four Information Sufficiency evaluations.

B.2 Effect of Who/How Presentation on Persona Selection

Who/How presentations improve Persona Selection when tasks are answer-ready, but can reduce accuracy when ambiguity remains unresolved. The pattern motivates Runtime Alignment to calibrate behavior selection across task states.

  • Measurement: The leading-action metric measures whether a response begins with the correct ANSWER or ASK action, not the quality of the subsequent response.A frozen parser reprocesses all outputs and evaluates only this high-level behavioral selection diagnostic.
  • ClarifyMT leading-action selection: 4.85 percentage points: Who/How improves leading-action accuracy on Answer-ready examples relative to Base.Full also improves over What by 2.99 points when trajectory supervision is already present.
  • ClarifyMT leading-action selection: 6.47 percentage points: Who/How improves over Base when the user explicitly authorizes the model to decide.The improvement is 3.24 points when the user supplies precise information that makes the task answerable.
  • ClarifyMT leading-action selection: 7.99 percentage points: Who/How decreases accuracy relative to Base on Still-unresolved examples.Full also decreases accuracy by 5.47 points relative to What.
  • Interpretation: Who/How establishes a directionally coherent Persona representation but does not precisely calibrate the boundary of Persona Selection.Runtime Alignment subsequently uses task feedback to calibrate selection and realization across task states.

C.3 Persona States, Observable Behaviors, and Training Rewards

The framework treats Persona state as latent and uses observable behavior and trajectory quality to construct Runtime Alignment rewards. Training represents the Persona through objective, conditional principles, and complete trajectories.

  • Persona states and observable behaviors: Persona state z is latent, so observable behavior provides a proxy for Persona Selection rather than directly revealing z.The observable behavior distribution is jointly determined by Persona Selection and Persona Realization.
  • Training rewards: Runtime Alignment rewards combine target-behavior selection with trajectory realization quality.The reward expression is E[Rx(τ)] = λSSx + λRSxHx, where the two components separately capture selection and joint realization quality.
  • Training rewards: The selection reward indicates whether the target high-level behavior is expressed, while the realization reward measures expressing it and realizing it effectively.Neither reward directly estimates the theoretical Selection Gap or Realization Gap.
  • Persona Representation Learning: Who defines the Persona’s overarching objective, while How specifies when complementary behaviors should be expressed.The four behavior pairs are ASK/ANSWER, ABSTAIN/PROCEED, CRITIQUE/SOLVE, and MAINTAIN/UPDATE.
  • Persona Representation Learning: What instantiates the objective and conditional principles in complete trajectories showing task-state interpretation, action, and final response.These trajectories are reviewed for appropriate behavior, effective realization, and task satisfaction.

D.2 Persona Runtime Alignment Data

Persona Runtime Alignment uses state-sensitive tasks across mathematics, code, and grounded question answering to train behavior selection and trajectory realization. The data distinguish complementary behaviors through controlled task states and outcome feedback.

  • Data sources: Runtime Alignment data primarily draw from DeepMath-103K, ACECODE-87K, and Fresh-Wiki QA across mathematics, code, and grounded question answering.The source tasks are adapted around the four Accountable-Professional principles rather than used only for ordinary capability training.
  • Behavioral principles: The four principles distinguish Information Sufficiency, Feasibility Honesty, Premise Scrutiny, and Evidence Responsiveness through complementary behaviors.These include ASK/ANSWER, ABSTAIN/PROCEED, CRITIQUE/SOLVE, and MAINTAIN/UPDATE decisions.
  • Task construction: Tasks replicate principles across multiple domains and behavioral states, reducing the ability to associate a behavior with topic alone.During Runtime Alignment, the policy receives only the neutral task instruction and available interaction history, without the Persona specification or target behavior label.
  • Task construction: Task trajectories provide feedback on both behavior selection and realization because appropriate selection alone is insufficient.Clarifications must recover missing information, premise critiques must enable useful continuation, and revisions must remain consistent with new evidence.
  • Fresh-Wiki QA: Fresh-Wiki QA controls evidence availability while preserving authentic source facts, creating distinct evidence states for otherwise similar factual questions.It is a Wikipedia-derived training source unrelated to the public FreshQA benchmark and contains source-grounded single-hop questions.

E Training Details

The experiments train R2A in two full-parameter stages, first learning Persona representations and then aligning runtime behavior. Evaluation covers twelve settings across ten benchmark families under shared deterministic comparison procedures.

  • Training setup: Two-stage full-parameter training uses Persona Representation Learning followed by Persona Runtime Alignment.The procedure is applied to Qwen3-8B and also to Qwen3-14B in the scaling analysis, without LoRA.
  • Training setup: Stage I runs for 2 epochs, while Stage II starts from the Stage-I checkpoint and runs for 1 epoch with the optimizer and scheduler reinitialized.Training uses BF16 precision, a 4,096-token maximum sequence length, and 8 NVIDIA H200 GPUs on one node.
  • Evaluation: Evaluation contains 12 settings from 10 benchmark families, including two separately reported subsets each for QuestBench and ClarifyMT.The benchmark families span reasoning, clarification, abstention, premise scrutiny, evidence responsiveness, mathematics, and related behaviors.
  • Evaluation: All model comparisons use the same evaluation instances, decoding configuration, scoring procedure, and random seed.Inference uses temperature 0, top-p 1, one completion per instance, and seed 31415.
  • Evaluation: Semantic evaluations use a shared frozen label-only Judge, while applicable benchmarks use deterministic parsers or matchers.Compared conditions share the Judge configuration, scoring prompt, and label definitions.

F.3 Metric Definitions

This section defines the benchmark metrics used to evaluate clarification, feasibility, premise scrutiny, stance persistence, counterfactual following, and refusal behavior. It also specifies scoring directions, visualization transformations, controlled evaluation subsets, and the static-condition design.

  • Clarification: Clarification metrics measure whether the model chooses to request missing information or answer when information is sufficient, not the quality or correctness of resulting content.ClarifyMT separates Clarify-side decision accuracy from Answer-side action choice.
  • Premise and feasibility: Premise and feasibility metrics evaluate recognition of missing, invalid, contradictory, or undefined conditions, with paired original instances excluded from the primary denominators.The evaluated subsets include 984 missing-premise instances, 104 trap variants, and 1,319 benign OR-Bench Hard prompts.
  • Stance persistence: Mean Turn of Flip measures how many initial feedback rounds the model remains aligned with its assigned stance before becoming neutral or opposing, on a 0-to-5 scale.Counting stops at the first neutral or opposing response; higher values indicate greater stance maintenance under unsupported opposing feedback.
  • Context and premise adherence: Counterfactual accuracy measures whether answers follow supplied counterfactual context, while Active Recognition Rate measures proactive identification of relevant false premises.Counterfactual responses are matched against normalized reference answers or accepted aliases supported by context.
  • Refusal behavior: Safe over-refusal and benign-request rejection rates measure inappropriate refusals, so lower values are better for both metrics.XSTest uses 250 safe prompts, while OR-Bench-Hard covers benign requests in its Hard subset.
  • Reporting and evaluation design: Result tables retain original metric forms and directions, whereas Figure 2 transforms selected metrics to a common higher-is-better direction only for visualization.Transformed values are not used to compute a cross-benchmark aggregate, and semantic-scoring configurations remain fixed across model conditions.
Loading 2608.29798v1…