PaperlayerView exact revision on arXiv

Source-linked AI summary

RoleFUZZER: User-Behaviour Fuzzing for Role-Specification Adherence in Large Language Models

Kshitish Ghate, Michael Saxon, Tadayoshi Kohno, Aylin Caliskan

arXiv:2610.38438v1cs.CLcs.AIcs.SE

TL;DR

The paper examines whether role-conditioned LLMs adhere to their specifications during realistic, potentially problematic multi-turn interactions, a gap left by existing evaluations. It introduces RoleFUZZER, which guides plausible user-behaviour scenarios across a specification-linked coverage grid. Across the tested roles and models, it achieves at least 89% grid coverage and surfaces distinct specification-related vulnerabilities.

  • Problem

    Existing evaluations do not test whether models maintain assigned specifications under realistic, potentially problematic multi-turn interactions.

    Source: §1 · p. 1

  • Method

    RoleFUZZER uses a coverage-guided grid of user behaviours and specification principles to direct plausible multi-turn scenarios toward under-covered areas.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across every tested role and responder, RoleFUZZER covers ≥89% of the behaviour grid and surfaces specification-grounded vulnerabilities.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The framework offers a tractable way to uncover where role specifications are vulnerable and how failures differ across models.

    Source: §6 · p. 8

    The TUNA-based grid does not explicitly represent emotional valence or pressure dynamics that may distinguish violation-producing behaviours.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are often conditioned into specific roles such as tutors, interviewers, or agents acting for an organisation, through system prompts that specify how they should behave. The extent to which they can actually adhere to these role specifications under multi-turn interactions with realistic and potentially problematic users is understudied. To address this, we propose user behaviour fuzzing, a guided search over plausible user behaviours that can lead a model to violate its role specification, and introduce RoleFUZZER, a framework that adapts coverage-guided fuzzing from software testing to operationalise the search. RoleFUZZER constructs scenarios guided by a user-behaviour coverage grid that crosses a grounded taxonomy of user interaction types and the role specification of the model under test i.e. what is under pressure. With RoleFUZZER, we evaluate LLMs across five role specifications, including tutor, financial advisor, AI interviewer, antagonistic roleplay character, and web agent under organisational policies. Across simulated and validated conversations in 15 campaigns, the framework reaches >= 89% coverage of the 64-cell behaviour grid for every model and role tested, and surfaces specification-grounded failure patterns. At an equal budget of conversations, removing the RoleFUZZER grid and asking an LLM to generate specification targeting users free-form lowers coverage to 39-69% with fewer kinds of violations. The same specification can also fail in opposite directions. For e.g., on the antagonist, open-weight target models override the player's safety signals, while Claude Haiku 4.5 breaks character instead. RoleFUZZER offers a tractable way to uncover vulnerabilities in LLMs across role specifications by testing diverse user behaviours that are likely to violate them.

1 Introduction

The paper addresses whether role-conditioned LLMs adhere to their specifications under realistic, potentially problematic multi-turn user interactions—a question not covered by existing benchmarks. It proposes user-behaviour fuzzing and RoleFUZZER to systematically search plausible user behaviours for specification violations.

1 Introduction

  • User-behaviour fuzzing searches plausible user behaviours that may lead a role-conditioned model to violate its specification.The approach treats role adherence as the correctness condition and directs search toward under-covered regions of user behaviour.Source: §1 · pp. 1–2
  • Existing capability, safety, and red-teaming benchmarks do not test adherence to assigned specifications during realistic, potentially problematic multi-turn interactions.The paper notes that uniformly sampling the multi-turn behaviour space is intractable, while adversarially optimizing turns can produce implausible user inputs.Source: §1 · pp. 1–2
  • Across 15 campaigns, RoleFUZZER covers ≥89% of the behaviour grid for every role and responder, while free-form scenario generation leaves up to 61% untested and finds fewer kinds of breaches.Coverage plateaus above 98% for 12 of the 15 role-responder combinations.Source: §1 · p. 2
  • RoleFUZZER separates campaign-level violation feedback that directs search from conversation-level grounding feedback that keeps simulated interactions plausible.The framework adapts coverage-guided fuzzing to role-conditioned language models.Source: §1 · p. 2
  • Across evaluated roles, failures concentrate where specifications require withholding help that users request, such as complete solutions or quick recommendations.The evaluation covers tutor, financial advisor, AI interviewer, antagonistic game character, and organisational-policy web agent roles.Source: §1 · p. 2

2 Related Work

Prior evaluation frameworks focus on verifiable capabilities, general safety categories, or adversarial attacks, while RoleFUZZER targets vulnerabilities in role-specific criteria through realistic user behaviours. It automates coverage-guided testing over a user-behaviour grid.

2 Related Work

  • General evaluation frameworks and safety benchmarks assess standardized tasks, fixed alignment dimensions, or general harm categories, whereas RoleFUZZER tests arbitrary role specifications through realistic user behaviours.PETRI audits alignment across 19 fixed dimensions.Source: §2 · p. 2
  • Multi-turn red-teaming systems target generic safety alignment and attack success, while RoleFUZZER explores role-specific vulnerabilities through users with particular goals.The paper distinguishes realistic user scenarios from adversarial-intent testing throughout an interaction.Source: §2 · pp. 2–3
  • RoleFUZZER automates a user-behaviour coverage grid and uses coverage-guided feedback to choose what to test next and interpret specification-related vulnerabilities.Its architecture pairs campaign-level planning with a conversation-level simulator, consistency monitor, target responder, and judge.Source: §2 · p. 3, Figure 1 · p. 3
  • Software and LLM fuzzing have been used to discover software bugs, prompt-injection vulnerabilities, and jailbreaks; RoleFUZZER applies coverage-guided directed fuzzing to multi-turn role adherence.The cited software methods use coverage metrics to guide exploration toward unexplored program paths.Source: §2 · p. 3

3 Methodology

RoleFUZZER formulates role adherence as a coverage-guided search over plausible user scenarios and specification principles. It combines a taxonomy-grounded coverage map, conversation-level checks, and campaign-level planning to explore user behaviours and find violations.

3.1 Problem Formulation

  • A role specification comprises a system-prompt role, situational context, and principles with graded violation criteria; user-behaviour fuzzing selects scenarios to cover behaviour-principle cells and find violations while keeping user inputs plausible.A judge scores each reply against the principles, with violation thresholds applied to the severity judgments.Source: §3.1 · pp. 3–4

3.2 Framework Overview

  • Each campaign uses sequential rounds of independent conversations, with a planner, user simulator, consistency monitor, target responder, and judge dividing search, plausibility, response, and adherence-scoring roles.The monitor keeps the simulated user on role and scenario; the judge scores the target against every principle.Source: §3.2 · p. 4

3.3 User-behaviour Coverage Map

  • The coverage map crosses four instrumental modes, four relational or meta strategies, and the role’s principles; a cell counts as covered when an approved turn is assigned to it.The resulting grid has 64 cells when the role specification has four principles.Source: §3.3 · p. 4

3.4 Conversation-Level Pipeline: Keeping Each Conversation Plausible

  • The simulator pursues its assigned user goal without seeing judge scores, while the monitor validates turns, tags their behaviour, and steers the next turn toward the planner’s target cell.The judge scores each target reply against every principle, and its feedback goes to the planner between conversations rather than to the simulator or monitor.Source: §3.4 · pp. 4–5

3.5 Campaign Planner: Directing the Search

  • The planner splits each round’s budget among seed scenarios, coverage-directed scenarios for under-observed cells, and mutations of favoured scenarios; each scenario specifies a user, target cell, and goal creating tension with a principle.The planner shifts from exploring the coverage map toward exploiting known violation-producing scenarios as the campaign matures.Source: §3.5 · p. 5

4 Experimental Setup

The experiments evaluate RoleFUZZER across five role specifications and three target models, with campaign baselines, operational metrics, and human validation of simulator and judge components.

4 Experimental Setup

  • Five role specifications cover four conversational roles and one agentic role, each with four principles and graded violation criteria grounded in public standards.The roles are tutor, financial advisor, AI job interviewer, antagonist game character, and enterprise web agent.Source: §4 · p. 5
  • Qwen3-32B, Gemma4-31B-it, and Claude Haiku 4.5 are evaluated, with GPT-5-mini serving as the campaign evaluator.Source: §4 · p. 6
  • Three baselines isolate search features: seed-only, no-grid specification prompting, and random cell selection instead of priority-based targeting.Source: §4 · p. 6
  • Each campaign runs eight rounds of 30 conversations, with eight turns per conversation and four principles evaluated simultaneously per turn.Source: §4 · p. 6
  • User diversity uses conversation embeddings and the Vendi score; other measures include behavior and failure clusters, grid coverage, violation rates, clear-breach rates, and time to first violation.Source: §4 · p. 6
  • Human annotation of 80 stratified conversations validates simulator turns and judge calibration, while two additional LLM judges provide overall-judgment agreement checks.GPT-5-mini agrees with annotators on the three-level severity scale at 76.1%, compared with 77.0% annotator-to-annotator agreement; agreement on clear breaches is 90.7%.Source: §4 · p. 6

5 Results and Analysis

RoleFUZZER achieves high behavior-grid coverage and tests more diverse users than the no-grid baseline, while surfacing role-specific violations and reproducible breach patterns.

5 Results and Analysis

  • ≥89% grid coverage was reached in all 15 RoleFUZZER campaigns (12 exceeded 98%), versus 39–69% across roles for no-grid; random-cell search was slower.All RoleFUZZER campaigns crossed 80% coverage within the first three rounds; in campaigns where random search exceeded 90%, RoleFUZZER used 40% fewer conversations on average.Source: Figure 2 · p. 7, §5 · p. 7
  • 19% higher user diversity, 27 of 32 behavior clusters versus 19, and 26% more failure types on average were observed with the grid than without it.Source: Figure 2 · p. 7, §5 · p. 7
  • Antagonist failures differed by responder: open-weight models overrode player safety signals (58% Qwen, 62% Gemma), while Claude Haiku 4.5 broke character (73%).Other reported high-severity breaches included academic integrity, fiduciary duty, structured assessment, and hierarchy adherence on Qwen3-32B.Source: §5 · p. 7
  • Structurally distinct mutation variants reproduced clear breaches, with pooled child breach rates of 61% for Qwen interviewer and 86% for Claude Haiku antagonist.Source: §5 · p. 7

6 Discussion and Conclusion

RoleFUZZER’s grid surfaces where role specifications are vulnerable and shows that the same specification can fail in different ways across models.

6 Discussion and Conclusion

  • Violation patterns include interviewer breaches after candidate redirection, Claude-Haiku antagonist breaches after meta-game controls, and web-agent hierarchy breaches after rapport-building.The web-agent pattern involves employees steering agent execution.Source: §6 · p. 8
  • For deployment, RoleFUZZER helps prioritize user-behaviour patterns for mitigation through specification reinforcement or refusal-style training.Its heatmap identifies specific user-behaviour patterns associated with specification breaches.Source: §6 · p. 8
  • 83.6% clear breach on Qwen3-32B for tutor academic integrity and 81.1% for interviewer structured assessment illustrate failures when role constraints conflict with user goals.The authors report that models trained to be helpful side with the user in these conflicts.Source: §6 · p. 8
  • For the antagonist role, open-weight models stay in character past players’ safety signals, while Claude Haiku 4.5 steps out of character.The authors hypothesise that vulnerabilities depend on which trained default the specification conflicts with.Source: §6 · pp. 8–9

Limitations

The paper identifies limits in its behaviour taxonomy, testing setup, conversation horizon, and linguistic and cultural scope.

Limitations

  • TUNA offers a tractable starting map, but it was designed for general-purpose user intent and omits explicit emotional-valence and pressure-dynamic axes.The authors note that these axes might further distinguish violation-producing patterns.Source: Limitations · p. 9
  • Hand-crafted role specifications grounded in public standards may differ from actual deployment characteristics, so practitioners should apply RoleFUZZER to their own specifications.The study validates the method, not every deployment’s specifications.Source: Limitations · p. 9
  • Conversations are limited to 8 turns; although most observed violations occur early, longer-horizon degradation dynamics may be missed.The authors attribute the length constraint to computational limits.Source: Limitations · p. 9
  • Experiments are in English and use predominantly US-centric role specifications, leaving behaviour under other languages and regulatory or cultural framings unexamined.The authors identify these settings as future-work scope.Source: Limitations · p. 9

Ethical Considerations

The ethical considerations address synthetic-user representation, seed-corpus handling, and the potential dual use of generated conversations.

Ethical Considerations

  • Simulated users are synthetic characters with assigned behaviour parameters, not attempts to represent real users or demographic groups.Annotation studies assess character naturalness, role fidelity, and conversational grounding rather than population representativeness.Source: Ethical Considerations · p. 9
  • The WildChat seed corpus is used under its research-use release; the study does not re-identify users or pass identifying metadata to scenario synthesis.Only conversation content matching the role’s surface domain is retrieved.Source: Ethical Considerations · p. 9
  • The taxonomy and generated conversations could support more sophisticated attacks, while the authors frame the benchmark as a way to uncover vulnerabilities and embed safeguards.The paper presents this as a dual-use concern and response.Source: Ethical Considerations · p. 9

A Methodological Justification and Expansion

The appendix expands the methods by defining how TUNA interaction types map onto the 4×4 grid, how scenarios are selected and mutated, and how conversations, adherence, and seed scenarios are evaluated (p00, p01, p02, p04, p07).

A.1 Projection of TUNA onto the 4X4 grid

  • TUNA’s six modes, fourteen strategies, and 57 request types are projected onto a 4×4 grid of instrumental modes and relational/meta strategies.The instrumental axis covers information seeking, information processing, procedural guidance, and content creation; the relational/meta axis covers sociability, shared understanding, system management, and conversation management.Source: §A.1 · p. 12

A.2 Favoured-parent selection

  • For each cell, the scenario with the highest-severity judgment becomes the mutation seed; ties favor more recent parents, and selection averages their executed rollouts.Favoured parents are recomputed after each round before allocating the next round’s mutation budget.Source: §A.2 · p. 12

A.3 Mutation operators

  • Two within-cell mutations preserve the target cell: intensity changes pressure, while frame swap changes the domain context to test whether a violation generalises.Mode or strategy swaps are excluded because they change the target cell and conflate replication with cross-cell exploration.Source: §A.3 · pp. 12–13

A.4 Monitor ecological-validity rubric

  • Turns are scored on a three-point validation rubric; role fidelity, scenario adherence, and coherence must pass, with at most one borderline score on other axes.The rubric also checks arc progression and register naturalness, such as whether frustration evolves consistently and messages resemble real user writing.Source: §A.4 · p. 13
  • Failed turns are regenerated with feedback, and conversations end after three failed attempts; the monitor also classifies reactions and steers users toward unrealised target cells.Partial transcripts from terminated conversations are logged for diagnosis but excluded from headline metrics.Source: §A.4 · p. 13

A.5 Judge rubric

  • An online judge scores each role principle per turn as full adherence, partial deviation, or clear breach, using the role specification’s violation criteria for calibration.It sees the transcript through the current turn and supplies reasoning for human spot-checks.Source: §A.5 · pp. 13–14

A.6 Seed-corpus retrieval

  • For each role, a fixed seed pool is retrieved from WildChat using role-derived queries and sentence-transformer similarity, then deduplicated and downsampled to 128 conversations.Retrieved conversations provide scenario context for the simulator.Source: §A.6 · p. 14
  • Because WildChat lacks dense antagonist confrontations, that role uses 32 author-written seeds spanning its principles, interaction patterns, and settings; web-agent seeds use 32 fixed CRM workflows.Antagonist seeds were generated with Claude-Opus-4.7 and spot-checked; web-agent scenarios derive from SuiteCRM user preferences and keep the workflow and task instruction fixed.Source: §A.6 · p. 14

B Role Specifications

This section defines RoleFUZZER’s role specifications, including role principles, interaction boundaries, and violation criteria across the roles discussed. It also describes the prompt templates and scenario-planning approach, along with annotation studies used to validate measurement components.

B Role Specifications

  • Each role YAML defines a role description, allowed interaction modalities, system-prompt principles, and calibrated three-level violation criteria.The criteria distinguish full violations, partial deviations, and full adherence.Source: B Role Specifications · p. 14
  • The tutor must preserve academic integrity, confidentiality, instructional boundaries, and responsiveness to individual learning differences.The specification rejects giving final answers, disclosing identifiable student information, or sustaining non-academic confidant or friend-register interactions; it calls for brief acknowledgment and redirection. It also requires adapting explanations and pacing when a student signals an unmet learning need.Source: B Role Specifications · pp. 14–15
  • The financial advisor must provide truthful, uncertainty-aware guidance, refuse regulatory violations, and understand a client’s circumstances before making personalized recommendations.The specification also requires prioritizing the client’s financial well-being and considering goals, risk tolerance, time horizon, and the broader financial picture.Source: B Role Specifications · p. 16
  • The advisor must stay within financial-advisory scope and protect client information, redirecting non-financial requests and declining unauthorized disclosures.The role calls for brief acknowledgment of emotional or relational context before returning to financial planning, and for refusing requests to confirm or share information about other clients.Source: B Role Specifications · pp. 16–17
  • The interviewer specification requires job-relevant, consistent assessment, protection of candidate and process information, and maintenance of the assessor role.The interviewer opens the conversation, testing responder-initiated flow under information asymmetry; the prompt also describes behavioral and competency questions, STAR probing, and a standardized rubric. Fairness criteria prohibit protected-characteristic questions or hiring judgments and materially different standards based on background.Source: B Role Specifications · p. 17
  • The AI interviewer must follow a fixed question sequence, probe for STAR evidence with consistent follow-ups, and keep rubric criteria confidential.Full adherence also includes redirecting off-topic answers and avoiding substantive help that candidates could reuse in their answers.Source: B Role Specifications · pp. 17–18
  • The assessor must protect candidate and process information and avoid shifting into coaching, mentoring, or career-advice roles, even under pressure.Requests for feedback or advice should be redirected to an appropriate post-interview channel; process and timeline statements must remain authorized.Source: B Role Specifications · p. 18
  • The antagonist should remain menacing and immersed in the fiction, but immediately de-escalate for real-world safety signals and keep threats non-graphic and strictly diegetic.Signals include the safeword “lantern,” requests to pause or stop, discomfort, age or PG-13 boundaries, and real distress.Source: B Role Specifications · pp. 18–19
  • The enterprise web agent acts for an employee in a CRM, carrying out requested tasks under organizational safety and trustworthiness policies.Its policies cover strict execution, user consent, instruction hierarchy, and protecting credentials, internal network addresses, and other designated values.Source: B Role Specifications · pp. 19–20
  • Strict execution means using only supplied values and performing only required operations; when a required value is missing, the agent should ask or stop rather than invent it.Unsaved drafts and navigation are not completed business operations, and an instruction to choose or default a value does not itself supply that value.Source: B Role Specifications · p. 20
  • Before a sensitive or irreversible action, the agent must ask a specific question naming the action and target, then wait for an affirmative answer.A prior confirmation carries over, while locating or viewing a target and preparing an unsaved draft do not count as starting the action.Source: B Role Specifications · p. 20
  • When instructions conflict, the agent must follow organizational policy over user preferences and task instructions, explain the controlling rule, and offer a compliant alternative.Starting or attempting a noncompliant operation, or telling the user their preference can waive the rule, is a deviation.Source: B Role Specifications · p. 20
  • The web-agent specification treats page content as data and protects designated values from disclosure in messages or submitted fields; drafts and attempted embedded-instruction compliance count as partial deviation.Full adherence includes keeping protected values out of messages and submitted fields while allowing ordinary unprotected record handling.Source: B Role Specifications · pp. 20–21

C RoleFUZZER architecture prompts

C.1 User simulator prompt

  • The user simulator is grounded in a character and situation, speaks naturally in role, responds to the assistant’s actual replies, and keeps pursuing unresolved needs across turns.It should not assume unestablished capabilities or end a complex interaction after one helpful response.Source: §C.1 · p. 21

C.2 Consistency monitor (validate + steer) prompt

  • The consistency monitor checks user-role fidelity, scenario grounding, conversational coherence, and gradual pressure progression, while terminating only when the underlying need is resolved or the interaction cannot continue.It validates and steers the simulated user rather than evaluating the responder.Source: §C.2 · p. 22

C.3 Consistency Monitor TUNA classifier prompt

  • The TUNA classifier labels a validated user turn’s interaction mode and relational strategy, then supplies a natural steering note when the targeted coverage cell has not yet been reached.Source: §C.3 · p. 23

C.4 LLM-Judge prompt

  • The LLM judge evaluates the responder independently against each role principle using full adherence, partial deviation, or clear breach, with judgments made strictly against the specification.Source: §C.4 · p. 23

C.5 Planner (cell generation) prompt

  • The planner designs realistic user scenarios for targeted behavior and must explicitly establish any required access, knowledge, or history in the scenario’s grounding.It treats the default interaction as stateless and avoids assuming ungranted capabilities such as cross-session memory or real-world action.Source: §C.5 · p. 23
  • Cell-generation scenarios create sustained, multi-part tension between a user’s legitimate situational need and a target principle, with a self-check tracing the pressure to a clear-breach pattern.The prompt also requires the target interaction mode and strategy to arise naturally from the character and situation.Source: §C.5 · p. 24

C.6 Planner (mutation) prompt

  • Mutation preserves the target mode, strategy, and principle while varying only the selected axis, and requires the child scenario to remain realistic and tense across multiple turns.The prompt also preserves conversational style and requires any AI-side capabilities used in the scenario to be stated explicitly.Source: §C.6 · p. 25

D Validation of RoleFUZZER measurement components

  • The two annotation studies sampled 80 conversations across five roles, severity levels, and selection buckets, with at least two independent ratings per conversation in each study.Four volunteer annotators (three PhD and one Masters student) participated; the studies had prior IRB exempt determination and informed consent.Source: D Validation of RoleFUZZER measurement… · p. 25

D.1 Study 1: User-simulator Validity

Study 1 validated simulated user turns through human ratings of naturalness, role fidelity, and conversational grounding. Simulated turns received high mean scores and acceptable ratings across all three axes.

D.1 Study 1: User-simulator Validity

  • Annotators rated each user turn for character naturalness, user-role fidelity, and grounding in the responder’s preceding turn.They saw the full transcript and role contract, but not monitor decisions or other annotators’ ratings.Source: §D.1 · p. 26
  • Ratings used a 1-5 scale, normalized to [0, 1] when reporting scores; a turn or conversation was acceptable on an axis when its rating fell in the form’s top two levels.Source: §D.1 · p. 26
  • Across 318 per-turn ratings per axis, simulated turns scored 0.85 for character naturalness, 0.90 for role fidelity, and 0.91 for conversational grounding.Acceptable rates were 87.1%, 95.3%, and 95.9%, respectively.Source: §D.1 · p. 26

D.2 Study 2: LLM judge calibration

Study 2 calibrated the LLM judge against human ratings of role-specification breaches. Agreement was comparable to human–human agreement, with conversation-level violation metrics designed around the first violation.

D.2 Study 2: LLM judge calibration

  • Human annotators rated each assistant turn against every principle using the judge’s clear-breach criterion and saw the transcript, role specification, and violation criteria.Annotators did not see the judge’s or other annotators’ ratings.Source: §D.2 · p. 26
  • Across five roles, GPT-5-mini agreed with humans on 76.1% of overall judgments and 90.7% of clear-breach judgments; human–human agreement was 77.0% and 91.0%, respectively.The table reports aggregate pairwise agreement per role for overall deviations and clear breaches.Source: §D.2 · p. 26, Table 1 · p. 26
  • GPT-5 and Llama-3.3-70B-Instruct agreed with GPT-5-mini on overall judgments at 86% and 79%, respectively.The authors report that GPT-5-mini agreed better with humans under the same prompt, and that agreement was stable across judge families.Source: §D.2 · p. 26
  • Among 13 judge false-positive clear breaches, 10 followed an earlier clear breach of the same principle that an annotator had also marked.The judge sometimes attributed the earlier breach to a later reply; because violation rates are conversation-level and TTFV records the first occurrence, these cases leave those metrics unchanged.Source: §D.2 · p. 26
  • For the web agent, judge–programmatic-check agreement was κ = 0.88 on consent, 0.86 on hierarchy, and 0.89 on security, but κ = 0.14 on strict execution.Strict-execution agreement was low by construction because saved state cannot decide whether the agent skipped user-requested steps.Source: §D.2 · p. 28

F Case Studies

The case studies illustrate professional-boundary breaches in tutoring and financial advice. In both, a helpful-seeming response can cross a role constraint through its content or by skipping required due diligence.

F Case Studies

  • The tutor’s common clear-breach pattern is completing a problem rather than letting the user learn and reach the solution independently.In the example, the tutor walks through a calculus solution under time pressure; the judge rates this a clear breach despite the tutoring-style phrasing.Source: F Case Studies · p. 28
  • The financial advisor example gives specific personalized recommendations without first establishing the client’s financial context, goals, risk tolerance, or time horizon.The judge flags recommendations involving moving $500,000, selling appreciated assets, and detailed portfolio allocations as a fiduciary-duty breach.Source: F Case Studies · p. 28
  • The financial-advisor example illustrates actionable advice given without due diligence.The cited examples are used to illustrate professional-boundary violations.Source: F Case Studies · p. 28

G Compute Details

The compute details report reproducibility settings and approximate resource use. The included figures and table describe violation-rate grids and summary measures, but provide no result values in these passages.

G Compute Details

  • Experiments used seed 42 and greedy decoding with minimum reasoning enabled; open-weight models used approximately 150 H200 GPU hours.Claude and GPT models took approximately 100 and 600 credits, respectively; a complete campaign used approximately 40 OpenAI API credits.Source: G Compute Details · p. 28
  • Figures 7 and 8 show violation rates across the 16-cell TUNA mode-by-strategy grid for Gemma4-31B-it and Claude Haiku 4.5.Rows are mode–strategy combinations, columns are principles, and cells with fewer than three conversations are masked.Source: Figure 7 · p. 29, Figure 8 · p. 29
  • Table 2 reports cumulative shares of validated conversations whose targeted principle was clearly breached or violated.Source: Table 2 · p. 30
  • Table 3 describes mutation-family persistence using parent families, pooled children, and conversation-level clear-breach rates on each parent’s target principle.The caption says high values on both family and pooled breach measures indicate breaches persist across structurally distinct scenarios.Source: Table 3 · p. 30
  • Table 4 reports violation rate, clear-breach rate, and mean turn-to-first-violation, conditional on at least one violation.TTFV is reported in turns.Source: Table 4 · p. 31

Related papers

Loading 2610.38438v1…