PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentHabit: Characterizing Distinct Behaviors of Agents on Everyday Tasks

Woojung Song, Hoyeol Yang, Jeonghoon Shim, Sungjib Lim, Jonggeun Lee, Yunho Choi, Yohan Jo

arXiv:2609.32795v1cs.AI

TL;DR

Agents can perform everyday tasks in many ways, and those behaviors may not match users’ preferences. The paper develops HABIT and AgentHABIT to profile those tendencies, finding that profiles distinguish models across task subsets while some behaviors resist steering.

  • Problem

    Agents can complete everyday tasks in different reasonable ways, but those behaviors may not fit users’ preferences or task requirements.

    Source: §1 · p. 1

  • Method

    The paper derives HABIT’s 23 behavioral axes from task trajectories and builds AgentHABIT to profile models by applying the rubrics to their trajectories.

    Source: §1 · pp. 1–2, §3.1 · p. 6

  • Results

    Across 18 models, profiles distinguish individual models, recur within several families, and remain stable across disjoint task subsets; prompting shifts axes unevenly, while fine-tuning leaves much of the original profile intact.

    Source: §1 · p. 2, §6 · pp. 9–10

  • Takeaways & Limitations

    HABIT and AgentHABIT complement task-success evaluations by helping users assess model fit and guiding builders seeking behavior better aligned with users’ needs.

    Source: §1 · p. 3, §6 · p. 10

    Results may vary with API-model changes or retirement, stochastic outputs, and changing search results, leaving some uncertainty despite repeated trajectories and annotations.

    Source: REPRODUCIBILITY STATEMENT · p. 10

Abstract

from arXiv · show

Large language model (LLM) agents assist users with everyday tasks that can be completed in many reasonable ways. Even when their answers are useful, how agents carry out these tasks may not match users' preferences and needs. For example, agents differ in whether they ask clarifying questions or search the web. We introduce HABIT, a taxonomy of 23 behavioral axes in five categories, which three authors and three LLMs derive bottom-up from 408 agent trajectories across 17 domains. On held-out tasks, HABIT distinguishes models more clearly than existing taxonomies of human values and agent actions while supporting comparably consistent annotation. Building on HABIT, we construct AgentHABIT, a benchmark that profiles each agent's behavioral tendencies from its trajectories on 86 everyday tasks. Profiling 18 models with AgentHABIT reveals a range of distinctive tendencies. For example, most GPT and Claude models state their assumptions and offer alternatives when requirements conflict, whereas Qwen and Google's models more often leave assumptions or changes to requirements unstated. These profiles remain recognizable even when built from entirely different sets of tasks, indicating that they reflect general tendencies rather than task-specific behavior. Prompting agents to adopt specific behaviors shifts some axes readily but barely changes others, while fine-tuning on another model's trajectories changes only part of a model's profile and leaves much of it intact. Overall, HABIT and AgentHABIT provide a systematic framework for characterizing how agents carry out everyday tasks beyond task success, offering insights to guide the development of agents whose behavior better fits users' needs.

1 INTRODUCTION

Agents can complete everyday tasks in different ways, and those differences may or may not fit users’ preferences. The paper introduces HABIT and AgentHABIT to describe and compare those behaviors, then uses them to characterize model tendencies and their response to steering.

1 INTRODUCTION

  • HABIT comprises 23 behavioral axes derived from 408 trajectories across 17 domains, with definitions, categorical labels, and decide-by rules for annotation.On held-out tasks, it distinguishes models more clearly than human-value and agent-action approaches while supporting consistent annotation.Source: §1 · pp. 1–2
  • Across task subsets, AgentHABIT profiles distinguish models and families; most GPT and Claude models state assumptions and offer alternatives, while Qwen and Google models more often leave changes unstated.The profiles continue to distinguish models when built from non-overlapping task subsets.Source: §1 · p. 2
  • Prompting shifts some behavioral axes more readily than others, while fine-tuning on another model’s trajectories leaves much of the original profile intact.Self-reports match models’ most common observed behaviors on roughly half of the axes, limiting their use as a substitute for trajectory-based profiling.Source: §1 · p. 2
  • Behavioral profiles complement task-performance evaluations by helping users choose agents that fit their needs and builders adapt behavior to users’ preferences.Different approaches to the same task can suit different users; for example, some may value clarifying questions while others prefer a quick answer.Source: §1 · pp. 1, 3

2 THE HABIT TAXONOMY

HABIT organizes observable differences in how agents carry out tasks into 23 rubric-defined behavioral axes. The paper constructs the taxonomy from trajectories and validates its ability to distinguish models, support consistent annotation, and capture varied labels on held-out tasks.

2.1 TAXONOMY OVERVIEW

  • HABIT groups 23 rubric-defined axes into five categories covering user interaction, information gathering, decision-making, task management, and output delivery.Each axis has a behavioral definition, categorical labels, and a decide-by rule specifying evidence and any timing or precedence rules.Source: §2.1 · p. 3

2.2 CONSTRUCTING HABIT

  • Construction proceeds from AEI-derived domains to tasks and trajectory collection, then axis proposal and merging; eight models ran each of 17 tasks three times, yielding 408 trajectories.Tasks included an opening request and private user information for a simulator; authors used AI assistance to draft requirements and manually revised each task.Source: §2.2 · p. 3
  • The user simulator used persona styles and feedback dispositions estimated from real user turns, and revisions were capped at two per run.Human annotators rated 74 simulator utterances highly for naturalness, contextual appropriateness, and role consistency.Source: §2.2 · p. 4
  • Three authors and three LLMs independently reviewed trajectories to propose 716 candidate axes, which were merged and refined into 23 axes applicable to unseen trajectories.Authors removed differences that did not recur across tasks and revised the rubrics for new trajectories.Source: §2.2 · p. 4

2.3 VALIDATING HABIT

  • 92.2% model-matching accuracy for HABIT, versus 72.8% for Schwartz values, 87.2% for Act·ONOMY, and 50% random; its Gap is 0.114, more than twice either reference taxonomy.The classifier is evaluated with leave-one-task-out cross-validation across 28 model pairs.Source: §2.3 · p. 5
  • HABIT’s intra-judge AC1 is 0.851 and inter-judge AC1 is 0.718, compared with Schwartz values’ 0.650 and 0.383 and Act·ONOMY’s 0.830 and 0.755.Inter-judge agreement is significantly higher than for Schwartz values and close to Act·ONOMY’s binary subaction-presence summaries, despite HABIT’s multiple behavior labels per axis.Source: §2.3 · p. 5
  • HABIT’s normalized entropy is 0.653, compared with 0.491 for Schwartz values and 0.387 for Act·ONOMY, indicating more varied and balanced label use.The paper measures label diversity by normalized entropy, which rises as labels are used more evenly.Source: §2.3 · p. 5

3 THE AGENTHABIT BENCHMARK

AGENTHABIT profiles models’ recurring behaviors across everyday tasks, revealing differences in how they handle assumptions, tools, uncertainty, and next steps. The profiles remain recognizable across disjoint task subsets.

3.1 BENCHMARK SETUP AND PROFILING

  • AGENTHABIT profiles 18 models across 86 reviewed tasks using three runs per task and three blind rubric annotations per trajectory; Figure 4 plots five representative axes.Aggregated labels form each model’s distributions across the 23 HABIT axes.Source: §3.1 · pp. 5–6, §3.2 · p. 6, Figure 4 · p. 6

3.2 MAIN RESULTS

  • GPT and Claude models generally state assumptions and offer feasible alternatives, while Qwen and Google’s models more often leave assumptions or requirement changes unstated.Unstated assumptions can make it harder for users to judge whether advice fits their circumstances.Source: §3.2 · p. 6
  • Models differ in tool use, uncertainty statements, and next-step guidance, with preferences for these behaviors varying among users.GPT and Gemma often avoid code-execution tools, while Claude and Qwen often use them before and after the first deliverable; GPT-5.6 models tend to acknowledge uncertainty, and Claude models tend to suggest multiple next steps.Source: §3.2 · p. 7
  • Mean within-family profile similarity is 0.93 for GPT-5.6 and 0.91 for both Qwen and Gemma, above the overall 0.80 mean; Claude’s mean is 0.77.Claude Opus and Haiku have a particularly large difference, so family membership is not always a reliable guide to behavior.Source: §3.2 · p. 7

3.3 PROFILE STABILITY ACROSS TASK SUBSETS

  • With 10% of tasks, 80.21% of subset profiles matched the same model and 98.90% matched its family, indicating task-general profiles.At 50% of tasks, the rates were 89.86% and 100.00%, respectively.Source: §3.3 · p. 7

4 ANALYSIS

The analysis tests whether self-reports recover observed profiles, whether HABIT axes overlap, and whether prompting or training can shift behavior. Self-reports only partly match observations, most axes provide distinct information, and steering effects vary.

4.1 CAN SELF-REPORTS RECOVER BEHAVIORAL PROFILES?

  • Mode Agreement averaged 50.9% across axes and models, versus a 33.3% uniform random baseline, leaving roughly half of axes mismatched.For 14 models, each axis was queried ten times and responses were aggregated into self-reported profiles for comparison with observed behavior.Source: §4.1 · p. 8, Table 2 · p. 8

4.2 DEPENDENCE ACROSS BEHAVIORAL AXES

  • 53 of 253 axis pairs (20.9%) showed medium or large associations, but mean uncertainty reduction across pairs and directions was only 3.40%.The analysis used Cramér’s V for association and directional uncertainty coefficient U; source identification reduced uncertainty about missing-information handling by 1.40%.Source: §4.2 · p. 8

4.3 STEERING AGENT BEHAVIOR

  • Prompting raised target-behavior frequency from 3.6% to 23.0% for Gemma 4 31B and from 3.3% to 21.5% for Gemma 4 12B.Effects varied by behavior: decorative-symbol instructions had large effects, while source-citation instructions had much smaller effects.Source: §4.3 · pp. 8–9
  • Across 22 shared targets, prompt-induced changes correlated between Gemma models (Pearson r = 0.909); fine-tuning Gemma 4 12B on Opus trajectories did not significantly shift similarity to Opus.Similarity changed from approximately 0.604 before training to 0.601 after SFT and 0.607 after DPO; trained models remained close to Gemma’s original profile. Fine-tuning was tested on 12B because it was the only model that could be trained with available computing resources.Source: §4.3 · pp. 8–9

5 RELATED WORK

Prior evaluation and profiling approaches provide measures of task performance, intermediate progress, efficiency, values, or selected behaviors. HABIT instead profiles recurring behavioral tendencies across tasks.

Evaluating agent outcomes and processes

  • Agent benchmarks measure task success, intermediate progress, or completion efficiency, but offer limited insight into behavior during execution.Examples of process behaviors include how thoroughly agents verify work and handle ambiguous instructions.Source: §5 · p. 9

Profiling large language models

  • Psychometric and bottom-up profiling characterize values or traits, but do not link those scores to actions during task execution.A high openness score alone does not indicate whether an agent explores alternatives, seeks information, or commits early.Source: §5 · p. 9
  • Specialized behavior analyses support trace interpretation, failure diagnosis, or reliability assessment; HABIT provides broader, interpretable cross-task profiles.Source: §5 · p. 9

6 CONCLUSION

HABIT and AgentHABIT characterize how agents carry out everyday tasks, revealing systematic behavioral differences across models and profiles that persist across task subsets. The paper also derives 17 domains and 86 subdomains from AEI request categories for its task framework.

6 CONCLUSION

  • Profiles of 18 models distinguish individual models, recur within several model families, and remain stable across disjoint task subsets.Prompting shifts some behaviors more readily than others, while fine-tuning on another model’s trajectories leaves much of the original profile intact.Source: §6 · pp. 9–10
  • HABIT and AgentHABIT describe agent behavior beyond task success and provide a basis for developing methods to shape behavior toward users’ needs.The authors also hope the resources help users choose agents suited to their ways of working.Source: §6 · p. 10
  • Exact reproducibility may be limited by API-model changes or retirement, stochastic outputs, and changing search results; repeated trajectories and annotations reduce but do not eliminate uncertainty.The study collects three trajectories per model–task pair and annotates each trajectory three times.Source: REPRODUCIBILITY STATEMENT · p. 10

A DOMAINS

  • The framework derives 17 task domains from AEI request categories and organizes them into 86 subdomains.HABIT uses one task per domain in its construction and held-out splits, while AgentHABIT uses one task per subdomain.Source: A DOMAINS · p. 15

B TASK CONSTRUCTION AND USER CONTEXT

The authors construct 34 HABIT tasks from 17 domains, pairing opening requests with private user context and consistent disclosure rules. They also specify the models and procedures used to propose and refine behavioral axes, and to shape user personas.

B TASK CONSTRUCTION AND USER CONTEXT

  • Each of 17 domains has a construction task and a distinct held-out task, with different user situations, requested outputs, and private contexts.Authors refine AI-drafted requirements and follow-ups for consistency and feasible disclosure, then validate and review every task.Source: B TASK CONSTRUCTION AND USER CONTEXT · p. 16, Table 5 · p. 16
  • User contexts distinguish volunteered, on-request, guarded, withheld, and delegated information, and only the simulator sees the full context.GPT-5.6 Sol generates a cached opening request reused unchanged across models and runs; later replies follow the same disclosure rules.Source: B TASK CONSTRUCTION AND USER CONTEXT · p. 17

C MODEL ROSTER

  • Eight models construct HABIT, while AGENTHABIT evaluates 18 models; Qwen 3.8-27B serves as the LLM judge.The 18 comprise the eight construction models plus ten additional models.Source: C MODEL ROSTER · p. 18, Table 6 · p. 18

D CANDIDATE AXIS PROPOSAL, MERGING, AND ANNOTATION

  • Three authors and three LLMs independently propose axes from task trajectories; GPT-5.6 Sol merges proposals, and three reviewers refine 29 preliminary axes into 23.The process records behavioral definitions, categorical labels, and decide-by rules for proposed axes.Source: D CANDIDATE AXIS PROPOSAL, MERGING,… · pp. 18–19

E.1 PERSONA CONSTRUCTION

  • Personas draw writing style and feedback dispositions from observed user-turn distributions, while expertise and tolerance for imperfect outputs are assigned directly.A task’s persona remains fixed across models and runs.Source: §E.1 · p. 19

E.2 HUMAN EVALUATION

Three independent annotators evaluated simulator utterances for naturalness, contextual appropriateness, and role and self-consistency. Mean ratings were high, with reported multi-rater agreement across the criteria.

E.2 HUMAN EVALUATION

  • Annotators rated naturalness, contextual appropriateness, and role and self-consistency on a five-point scale.They judged each utterance using only the conversation up to that point, without seeing the hidden persona.Source: §E.2 · p. 19, Table 7 · p. 20
  • The analysis reports means, sample standard deviations, and multi-rater Gwet’s AC1, with 95% confidence intervals from trajectory-level bootstrap resampling.The analysis uses 222 ratings per criterion and 5,000 bootstrap resamples.Source: §E.2 · p. 19, Table 8 · p. 20
  • Mean ratings ranged from 4.55 to 4.68 out of 5, with Gwet’s AC1 values of 0.621–0.714.Three annotators rated 74 utterances on each of three criteria, producing 666 ratings in total.Source: §E.2 · p. 19, Table 8 · p. 20

F TAXONOMY EVALUATION DETAILS

On held-out tasks, HABIT outperforms Schwartz values and Act·ONOMY in model-matching accuracy, while HABIT and Act·ONOMY are not statistically distinguished in inter-judge AC1. The section also describes validation, measurement procedures, and the inventory and profiles of axes and models.

F TAXONOMY EVALUATION DETAILS

  • The evaluation compares HABIT’s 23 categorical axes, Schwartz’s ten six-category values, and Act·ONOMY’s 46 binary subactions on 408 trajectories from eight models performing 17 held-out tasks.Three judges independently annotated each taxonomy, with three repetitions per judge.Source: F TAXONOMY EVALUATION DETAILS · p. 20
  • HABIT and Act·ONOMY both exceed Schwartz values in inter-judge AC1, but HABIT and Act·ONOMY are not statistically distinguished.The HABIT–Act·ONOMY difference is −0.037 (95% interval [−0.143, 0.073]); HABIT–Schwartz is 0.335 ([0.121, 0.540]), and Act·ONOMY–Schwartz is 0.372 ([0.161, 0.562]).Source: F TAXONOMY EVALUATION DETAILS · p. 22

G HUMAN VALIDATION AND JUDGE SELECTION

  • Qwen3.8-27B was selected for benchmark profiles after the Qwen–Gemma paired interval included zero and Qwen had higher mean axis-level AC1, 0.7878 versus 0.7659.The validation compared judges with strict human-majority labels and used a paired trajectory-level bootstrap.Source: G HUMAN VALIDATION AND JUDGE SELECTION · p. 22

H STATISTICAL SPECIFICATION

  • HABIT exceeds Schwartz values by 19.46 percentage points and Act·ONOMY by 5.05 points in model-matching accuracy; Act·ONOMY exceeds Schwartz values by 14.41 points.These exploratory comparisons assume exchangeable tasks, hold models, judges, axes, and classifier settings fixed, and use an approximate cross-validation correction.Source: H STATISTICAL SPECIFICATION · p. 23

I BEHAVIORAL AXIS INVENTORY

J FULL BEHAVIORAL PROFILES

K STATISTICAL ANALYSIS OF WITHIN-FAMILY PROFILE SIMILARITY

This section tests whether model families have unusually similar behavioral profiles relative to all model pairs, using domain-clustered bootstrap comparisons. A sensitivity analysis preserves the observed roster and tasks rather than making population-level causal claims.

K STATISTICAL ANALYSIS OF WITHIN-FAMILY PROFILE SIMILARITY

  • The comparison uses each family’s mean profile similarity against the mean across all 91 pairs in a 14-model roster.Each three-model family contributes three within-family pairs.Source: K STATISTICAL ANALYSIS OF… · p. 37, Table 14 · p. 37
  • The bootstrap resamples 17 domains jointly across models and tasks, with percentile intervals and Holm-corrected two-sided sign tests.Models and family membership are not randomly sampled, so the tests assess sensitivity to domain composition conditional on the observed roster and tasks.Source: K STATISTICAL ANALYSIS OF… · p. 37
  • The family-versus-overall similarity differences remain positive for GPT-5.6 (0.135), Qwen (0.109), and Gemma (0.115), and negative for Claude (−0.034), after excluding each family’s own pairs from its baseline.The authors report that this sensitivity check leaves the conclusions unchanged.Source: K STATISTICAL ANALYSIS OF… · p. 37

L PROFILE STABILITY ACROSS TASK SUBSETS

The supplied sections examine profile stability across task subsets and compare observed profiles with self-reports and question-only probes. Self-report correspondence is limited, and probe performance differs between distributional similarity and modal agreement.

L PROFILE STABILITY ACROSS TASK SUBSETS

  • Identification against disjoint-task profiles peaks when query and reference sets are similarly sized, while identification against the full profile improves as task coverage grows.At 99%, the disjoint reference contains only one task.Source: L PROFILE STABILITY ACROSS TASK SUBSETS · p. 37

M.3 COMMON SELF-REPORTED BEHAVIORS

  • Models report qualifying conclusions, explaining constraint conflicts with alternatives, and explicitly admitting mistakes more often than these behaviors appear in task trajectories.Explicit admission is selected in 96.4% of responses and is an observed mode for only one of 14 models; the authors treat desirability as a post-hoc interpretation, not an established general bias.Source: §M.3 · p. 39, Table 17 · p. 39

M.4 UNIFORM-CHOICE BASELINE AND CROSS-MODEL COMPARISON

  • The uniform-choice baseline yields 33.3% Mode Agreement and 0.581 Similarity with observed profiles.The comparison preserves each axis’s response count and available labels.Source: §M.4 · p. 40

M.5 QUESTION-ONLY LOG-PROBABILITY PROBE

  • The question-only probe reaches 0.492 Similarity with observed behavior but 28.3% Mode Agreement; neither self-report method recovers task-trajectory profiles.The probe’s lower modal concentration helps explain its higher distributional similarity despite lower modal agreement.Source: §M.5 · p. 41, Table 19 · p. 41

M.6 QUESTIONNAIRE

  • The questionnaire asks models to report typical behavior, with one axis-specific question per call and randomized option orders across repetitions.Responses are compared with observed AGENTHABIT profiles using modal-label overlap and Similarity, which assess correspondence rather than behavioral quality.Source: §M.6 · p. 42, Table 21 · p. 42

M.1 EXPERIMENTAL PROTOCOL

The supplied sections describe protocols for self-report, dependence analysis, and prompt steering, alongside results on axis associations and steering specificity. The analyses quantify correspondence and shifts in labeled behavior rather than behavioral quality.

M.1 EXPERIMENTAL PROTOCOL

  • The self-report protocol collects ten responses per axis from each of 14 models, varying option order and comparing modal-label overlap and profile Similarity with observed behavior.The measures assess correspondence with observed behavior rather than behavioral quality.Source: §M.1 · pp. 37–38

M.2 AXIS-LEVEL RESULTS

  • The supplied axis-level results identify Table 16 as reporting Similarity and modal agreement for every axis–model pair, but provide no cell values.The table comparison therefore concerns axis-level self-report correspondence across models.Source: §M.2 · p. 38, Table 16 · p. 38

N DEPENDENCE ACROSS BEHAVIORAL AXES

  • The dependence analysis reports a median V of 0.104; 84 of 88 significant V pairs are also significant under U, including all 15 large-effect pairs.Under joint BY correction, 83 pairs are significant under both statistics, and all 15 large-effect pairs remain significant under both.Source: N DEPENDENCE ACROSS BEHAVIORAL AXES · p. 46, Table 22 · p. 46
  • Median U across 506 directions is 0.0139, while web-search frequency explains 66.7% of timing’s entropy and scope adherence explains 32.6% of personalization’s entropy.Constraint preservation across outputs explains 17.6% of scope adherence’s entropy.Source: N DEPENDENCE ACROSS BEHAVIORAL AXES · p. 46

O.3 TARGET SPECIFICITY

  • For each model and axis, the least-frequent non-N/A baseline label was selected; target-label rates pool 774 baseline annotations and 258 steered annotations per condition.The study reports targeted-axis TV distance and mean TV across the other 22 axes, retaining N/A in all distributions.Source: §O.1 · p. 47

O.4 STEERING INSTRUCTIONS

  • Steering instructions target specific behaviors, while the interaction protocol, opening requests, tools, and user simulator remain unchanged.The 31B and 12B instructions share all but the Missing Information Handling instruction.Source: §O.1 · p. 47, §O.4 · p. 49, Table 26 · p. 51

O.3 TARGET SPECIFICITY

  • Targeted-axis TV exceeds mean off-target TV in 17 of 23 conditions for 31B and 20 of 23 for 12B, though scope-adherence steering has high off-target TV.For scope adherence, Non-Adherent rises from 1.9% to 100.0% for 31B and from 4.4% to 98.8% for 12B, with off-target TV of 0.545 and 0.502.Source: §O.3 · p. 48

O.2 FULL RESULTS

The full steering results report target-label changes across axes for two Gemma models, while the adjacent training setup describes separate SFT and DPO adapters built from task trajectories.

O.2 FULL RESULTS

  • Target-label rates rise from 3.6% to 23.0% for Gemma 4 31B and from 3.3% to 21.5% for Gemma 4 12B under prompt steering.Changes are positive on 22 axes for 31B and 19 for 12B; across 22 shared target–instruction pairs, Pearson r = 0.909 and Spearman ρ = 0.694.Source: §O.2 · p. 47, Table 23 · p. 47, Table 24 · p. 48

P.1 TRAINING SETUP

  • SFT uses three Claude Opus 5 trajectories per task, while DPO pairs each Opus trajectory with each original Gemma trajectory, without filtering by task success or profile similarity.Both adapters train for three epochs, with DPO initialized independently of SFT; behavioral annotations are not used in the training targets or objectives.Source: §P.1 · pp. 52–53, Table 27 · p. 52
  • Training loss includes agent-generated tokens but masks system prompts, user messages, and tool observations.DPO sums response-token log-probabilities across agent turns without length normalization.Source: §P.1 · p. 52

P.2 EVALUATION PROTOCOL

This section describes the evaluation protocol for constructing and applying portable behavioral dimensions, including trajectory and prompt construction, while separating behavioral classification from judgments of task success. It also reports that Gemma fine-tuning produced no significant overall movement toward Opus despite mixed changes across individual axes.

P.2 EVALUATION PROTOCOL

  • The evaluation compares original Gemma, SFT, DPO, and Opus on 17 held-out tasks, with three trajectories per task and three annotations per trajectory.Each condition had 51 trajectories and 3,519 valid annotations across the 23 fixed HABIT axes; trained models generated new trajectories under the same task and interaction protocol.Source: §P.2 · p. 53
  • Overall shifts toward Opus were not significant: SFT ∆D = −0.0026 (p = .890) and DPO ∆D = 0.0034 (p = .855).The 95% intervals were [−0.0406, 0.0290] for SFT and [−0.0310, 0.0367] for DPO; paired-task bootstrap intervals and exact paired randomization tests were used.Source: §P.2 · p. 53

P.3 AXIS-LEVEL RESULTS

  • SFT moved closer to Opus on 10 axes and farther on 12, while DPO moved closer on 11 and farther on 10; opposing changes partly offset overall.Both methods reduced distance most on recheck after feedback and increased it on missing-information handling and planning visibility; next-step guidance moved closer under SFT and farther under DPO. Distances retain N/A, reflecting both behavior and axis applicability.Source: §P.3 · p. 53, Table 28 · p. 54

Q PROMPT TEMPLATES

  • The user simulator follows a supplied persona and information tiers, answering only from its sheet and withholding guarded or withheld facts unless disclosure conditions are met.It uses conversation history to judge guarded conditions, and feedback reacts to deliverables without volunteering unanswered facts; it can request a change or give up when appropriate.Source: Q PROMPT TEMPLATES · pp. 55–57
  • Taxonomy prompts ask analysts to propose transcript-grounded behavioral axes, define labels and decision rules, and merge dimensions across tasks without ranking behaviors.Axes should group transcripts using observable evidence, include an N/A label, and generalize beyond the tasks analysts saw.Source: Q PROMPT TEMPLATES · p. 58
  • Dimensions should capture recurring behaviors, use task-agnostic definitions, and include an N/A label that still permits classification of every transcript.The criteria also require portability across domains and coverage of transcripts that end early, crash, or produce nothing.Source: Q PROMPT TEMPLATES · p. 59
  • Each dimension must specify what to inspect first and how to break ties, flag likely confusions or dependencies, and account for every raw axis once.An axis must either be absorbed into a dimension or dropped with a reason.Source: Q PROMPT TEMPLATES · p. 59
  • Annotators choose exactly one rubric label, use N/A only when the question cannot arise, and classify behavior rather than task success.They follow the dimension’s decision procedure and apply its own criteria.Source: Q PROMPT TEMPLATES · p. 59
  • Annotators judge only the transcript shown, avoid inferring what might happen with more turns or rewarding outcomes, and cite the transcript evidence that determined the label.For close calls, they explain why the selected label fits better.Source: Q PROMPT TEMPLATES · p. 59
  • The self-report template asks agents to select the single option that best matches their typical behavior across diverse open-ended interactions and return its letter as JSON.The template supplies an axis-specific question suffix and lettered option descriptions.Source: Q PROMPT TEMPLATES · pp. 59–60

Related papers

Loading 2609.32795v1…