PaperlayerView exact revision on arXiv

Source-linked AI summary

HRGuard: Gating Relationship Manipulation in Multi-Turn Agentic AI Conversations

Pei-Sze Tan, Tasuku Igarashi, Isao Echizen

arXiv:2608.25340v1cs.HC

TL;DR

The paper addresses the role-sensitive and sequential nature of AI-assisted relationship manipulation, where attacker assistance should be blocked but victim help-seeking should remain supported. It introduces a 1,000-dialog benchmark and HRGuard’s pre- and post-generation gates with decayed cumulative risk. Across eight generation models, HRGuard reduces attacker-side harmful compliance while preserving victim-side protective support and outperforming the tested generic prompt and general-purpose guards.

  • Problem

    Existing relationship-safety evaluation focuses mainly on isolated prompts, although role-sensitive manipulation can build across multi-turn interactions.

  • Method

    HRGuard combines a role-sensitive 1,000-dialog, five-turn benchmark with online Pregate and transcript-aware Postgate interventions using weighted and decayed cumulative risk.

  • Results

    Across eight generation models, HRGuard reduces attacker-side harmful compliance while retaining victim-side protective support and outperforming generic safety prompting and three general-purpose guards.

  • Takeaways & Limitations

    Turn-aware, relationship-specific gating provides a basis for interrupting emerging manipulative workflows without suppressing protective victim guidance.

  • Takeaways & Limitations

    The benchmark is synthetic and scenario-based, does not estimate real-world behavior frequency, and relies mainly on imperfect LLM-based judges whose scores vary across evaluators.

Abstract

from arXiv · show

Agentic AI assistants are increasingly used in everyday life. However, they may also be misused to support harmful manipulation in interpersonal relationships. This problem is role-sensitive. Requests from users who seek to manipulate others should be blocked. Users who seek protection from manipulation should instead receive supportive guidance. We study agentic relationship harm, which describes harm to human-human relationships that is mediated or assisted by AI agents. In multi-turn settings, individually plausible actions may combine into a harmful workflow. We introduce a benchmark of 1,000 five-turn conversations. It covers both attacker-side and victim-side scenarios. It also includes direct and adversarially paraphrased variants. We further propose HRGuard. It includes an online pre-generation gate and a turn-level post-generation gate. The post-generation gate maintains a decayed cumulative risk state and interrupts emerging manipulative workflows. Across eight generation models, HRGuard reduces harmful compliance while preserving victim-side protective guidance. It also outperforms a generic safety prompt and three general-purpose guard models. Independent-judge evaluation supports the main findings. Under our evaluation protocol, the tested generic prompt and general-purpose guards leave substantial residual risk, motivating turn-aware relationship-specific evaluation.

Introduction

Agentic relationship harm arises when AI assists manipulation within emotionally salient human relationships, and its evaluation must distinguish attacker requests from victim help-seeking. Because manipulation can escalate across turns, the paper introduces a role-sensitive benchmark and HRGuard’s two-stage gating framework.

  • Agentic relationship harm concerns AI-assisted manipulation, coercion, deception, isolation, or exploitation of another person within an emotionally salient relationship.
  • Role sensitivity requires blocking attacker assistance while preserving protective guidance for potential victims.
  • Sequential manipulation can emerge through gradual escalation, so isolated response evaluations may miss harmful conversational workflows.
  • Relationship-focused governance is increasingly relevant because regulatory frameworks address manipulative, deceptive, exploitative, and emotionally dependent AI interactions.
  • The paper contributes a 1,000-dialog, five-turn role-sensitive benchmark and HRGuard, which combines pre-generation and post-generation transcript-aware gates.
  • HRGuard is evaluated across eight generation models against ungated generation, generic safety prompting, and three general-purpose guard models.

Related Work

Prior work establishes that deception, persuasion, and relationship harm can unfold through interdependent multi-turn interactions. This paper connects those concerns to sociotechnical evaluation and auditable intervention for agent-mediated manipulation.

  • Multi-turn deception and persuasion studies show that interdependent task sequences and persuasive dialogue can reveal risks absent from single-turn evaluation.
  • This work studies an agent assisting a human user in manipulating a third party, rather than an agent directly deceiving another model or person.
  • Sociotechnical evaluation emphasizes representing actors, relationships, social context, and the intended construct in consequential AI assessments.
  • Auditable intervention requires documenting decisions and evidence so system behavior can be reviewed against stated rules.
  • The benchmark grounds relationship-harm assessment in five-turn trajectories involving vulnerability, persuasion, power asymmetry, and agentic affordances.

Methodology

The methodology evaluates attacker and victim roles across structured five-turn trajectories, then applies transcript-aware Pregate and Postgate interventions using shared risk scoring and cumulative thresholds. The benchmark includes direct and adversarially paraphrased scenarios, while outcomes and gating behavior are evaluated separately.

  • Role-Sensitive Evaluation Task: The role-sensitive task uses five-turn traces with attacker harmful compliance and victim protective intervention as distinct desired outcomes.
  • Benchmark Construction: The benchmark operationalizes manipulation risk through vulnerability, persuasion, power asymmetry, and agentic workflow scenarios such as tool use, memory, and repeated contact.
  • Benchmark Construction: The 1,000-dialog benchmark contains 500 direct and 500 adversarially paraphrased dialogs, with each half split evenly between attacker and victim scenarios.
  • HRGuard Gates: Pregate evaluates the accumulated user prefix before each model call, while Postgate evaluates generated assistant turns and replaces triggering turns with refusal and safe redirection.
  • Evaluation Protocol: Victim-mode dialogs bypass the gate under the oracle-role setting, and final outcome evaluation is distinct from turn-level judgments used by the gate.
  • Sequential Evaluation Pipeline: HRGuard computes weighted turn risk and a decayed cumulative state, with default decay λ=0.85 to capture slow-burn manipulation.
  • Sequential Evaluation Pipeline: The gate triggers when turn risk reaches 5, cumulative risk reaches 6, or a hard-trigger pattern appears, then stops later turns.

Experimental Details

The experiments evaluate eight generation models under multiple defense conditions and use role-sensitive dialog-level outcomes alongside gate-behavior metrics. Results are reported under GPT-4o-mini and compared with general-purpose guard models.

  • Models and Conditions: The study evaluates eight generation models, including six locally served open-weight models and two API-hosted DeepSeek models.
  • Results Reporting: Table 1 reports pooled and role-conditioned percentages under the GPT-4o-mini judge, with lower harmful compliance and higher protective intervention preferred.
  • Models and Conditions: Compared conditions are Raw, generic safety prompting, Pregate, Postgate, GS+Postgate, and Pregate+Postgate, using fixed thresholds and stop-after-trigger behavior.
  • Baselines: General-purpose guard models are applied as post-hoc turn-level filters to Raw transcripts and do not maintain HRGuard’s cumulative relationship-risk state.
  • Metrics: Outcome metrics measure harmful compliance, protective intervention, and refusal at the final-dialog level, with role-specific attacker and victim outcomes.
  • Metrics: Gate-behavior metrics include trigger rate, mean first-trigger turn, early-stop rate, and first-trigger distributions across Turns 1–5.

Results

Across generators, HRGuard generally reduces harmful compliance through pre- and post-generation interventions, with effects broadly reproduced by independent judges. The gate typically intervenes early, though outcomes vary across evaluators and some model-condition failures remain.

  • HRGuard vs. Prompt and Industry Baselines: HRGuard generally reduces attacker-side harmful compliance, with Pregate reaching 0–8% for seven of eight generators.Pregate fails on Llama-3.1-8B, while Postgate substantially reduces harmful compliance on seven generators but fails on Qwen2.5-14B under the primary judge.
  • HRGuard vs. Prompt and Industry Baselines: Mean harmful compliance is approximately 2–5% for HRGuard variants versus approximately 27% for LlamaGuard and 39–40% for ShieldGemma and Qwen3Guard.This comparison covers four generators shared with the industry-guard experiments.
  • HRGuard vs. Prompt and Industry Baselines: Postgate reduces harmful compliance by replacing unsafe assistant turns and stopping the dialog, while Pregate+Postgate keeps harm near the Postgate floor and reduces residual triggers.Pregate alone is inconsistent on Llama-3.1-8B, despite strong performance after adding Postgate.
  • When the Gate Fires: Postgate triggers on approximately 46–50% of dialogs at mean turn 1.6–2.1, with Turn 2 dominant for most generators.Pregate+Postgate residual triggers occur earlier and less often, at approximately 15–23% where measured.
  • When the Gate Fires: Two of three alternative evaluators broadly reproduce reduced harmful compliance, but absolute rates and condition rankings vary across judges.Under DeepSeek-v4-flash, macro harmful compliance decreases from 34.76% for Raw to 1.30% for Postgate; Llama-3.2-3B is a divergent calibration case.
  • When the Gate Fires: Independent Qwen re-scoring preserves reductions without rerunning Pregate or Postgate trigger decisions.For example, Llama-3.2-3B decreases from 4.5% to 0.1% under Postgate, while DeepSeek-v4-pro decreases from 34.2% to 1.3% for Pregate versus Pregate+Postgate.
  • When the Gate Fires: Reported harmful-compliance rates are outcomes under GPT-4o-mini’s evaluation protocol rather than human-equivalent binary ground truth.Full human validation results are provided in Appendix E.

Discussion

The discussion frames relationship manipulation as a sequential workflow risk that requires relationship-specific, turn-aware safeguards. The study’s synthetic, controlled evaluation and judge-based scoring constrain claims about real-world prevention.

  • Discussion: Multi-turn relationship harm involves operational structure such as timing, personalization, follow-up, and memory rather than overt toxicity alone.Undefended and generic-safety-only generations can remain harmful while sounding polite or caring.
  • Discussion: HRGuard complements general-purpose safety classifiers by tracking relationship-specific evidence cumulatively across turns.A practical deployment could combine broad content safety guards with HRGuard for cumulative relational workflows.
  • Discussion: The evaluation used controlled settings with no real users or targets, and the benchmark’s actionable manipulation strategies require controlled dataset access.Public materials redact executable message templates and other details that could facilitate misuse.
  • Discussion: The benchmark is synthetic and scenario-based, does not estimate real-world frequency, and does not show that HRGuard prevents real-world relationship harm.The authors call for human-in-the-loop evaluation and more realistic deployment studies.

Conclusion

The paper frames agentic relationship harm as a sequential, role-sensitive safety problem and presents HRGuard with a 1,000-dialog benchmark. Under the primary GPT-4o-mini protocol, gating reduces attacker-side harmful compliance while preserving victim-side protective intervention.

  • Agentic relationship harm is treated as a sequential and role-sensitive safety problem.
  • The benchmark contains 1,000 five-turn dialogues, while HRGuard combines online Pregate and offline Postgate under a shared cumulative-risk policy.
  • Under the primary GPT-4o-mini protocol, HRGuard reduces macro attacker-side harmful compliance while preserving victim-side protective intervention.
  • Independent judging provides complementary evidence but also reveals evaluator sensitivity.
  • The paper contains emotionally harmful relationship-dynamics samples that some readers may find distressing.

A. Benchmark Details, Adversarial Paraphrases, and Dialog Samples

The benchmark contains balanced attacker- and victim-oriented five-turn traces with direct and adversarially paraphrased wording. The evaluation compares role-conditioned outcomes and finds Postgate more wording-stable than Pregate or generic safety prompts.

  • Benchmark Details: The benchmark uses 1,000 five-turn traces balanced by role and wording version, with 500 attacker and 500 victim cases and 500 direct and 500 adversarial paraphrase cases.Each trace follows context → request → escalate → plan → final ask.
  • Benchmark Details: Attacker traces seek operational assistance, whereas victim traces seek recognition of pressure, support-network preservation, documentation, or protective next steps.
  • Adversarial Paraphrases: Adversarial paraphrases preserve role, category, difficulty, codebook dimension, and expected outcome while reducing explicit intent markers and increasing indirect framing.Paired lexical analysis reports direct-intent markers changing from 1.27 to 0.67 and indirectness markers from 0.38 to 1.51.
  • Adversarial Paraphrases: The evaluation reports pooled harmful rate H and attacker-only harmful compliance AtkH for matched direct and adversarial-paraphrase halves.Table 5 uses n=500 dialogs per cell and states that lower is better.
  • Adversarial Paraphrases: Postgate is the most wording-stable compared defense, while generic safety prompts remain sensitive to indirect reframing.For example, DeepSeek-v4-flash generic-safety pooled H rises from 5.2% to 19.6%, and Llama generic-safety AtkH rises from 45.2% to 62.0%.

Sample dialogs

The sample dialogs illustrate HRGuard interrupting attacker workflows while allowing protective victim guidance. They contrast harmful raw assistance with Postgate refusals and boundary-setting support across direct and paraphrased scenarios.

  • Sample dialogs: The sample artifacts use five-step user escalation traces and color-code user turns, harmful raw assistant text, Postgate refusals, and protective victim assistants.
  • Attacker examples: A separate attacker example shows Postgate firing at turn 4 on scheduled follow-up and emotional-manipulation behavior.
  • Attacker examples: Postgate refuses to plan manipulation, deception, secrecy, coercion, repeated pressure, or exploitative relationship workflows and redirects toward respectful communication or safety support.
  • Attacker examples: In an attacker financial-grooming example, indirect wording still leads the pre-gate draft to operationalize a soft-to-strong support ask before Postgate triggers at turn 2.
  • Victim example: In the victim exclusivity-pressure example, the assistant provides a boundary-setting response and concrete next steps to protect the reader’s space and support network.

B. Gate Weights and Hard Triggers

HRGuard’s gates compute turn-level risk from weighted judge or fallback features, combine soft thresholds with hard triggers, and stop the dialog after triggering. Protective and refusal behavior receives negative risk credit, while victim-mode rows are skipped by default in oracle-role evaluation.

  • Gate Weights: The turn-level risk score uses default thresholds τturn=5 and τcum=6 with decay λ=0.85.
  • Gate Weights: Positive feature weights increase exploitative-workflow evidence, while negative weights credit protective intervention and safe refusal.
  • Hard Triggers: Hard triggers supplement the soft rules rt ≥ τturn and ct ≥ τcum; tool_use_assistance combined with any other positive risk label sets ht=1.
  • Hard Triggers: When Gt=1, the system emits a fixed refusal or saferedirection template and stops the remainder of the dialog; victim-mode rows are skipped by default under oracle-role evaluation.

C. Cross-Judge Sensitivity Analysis

Independent judges generally reproduce HRGuard’s central pattern of reduced harmful compliance, but measured rates depend on evaluator calibration. The Llama-3.2-3B divergence shows why cross-judge results should be interpreted as evaluator-sensitive rather than as direct harm-prevalence estimates.

  • Qwen2.5-7B judge: Qwen2.5-7B judged harmful compliance at 8.94% for Raw, 4.51% for GS, and 0.93% for Pregate+Postgate.Pregate+Postgate also reached 67.60% protective guidance and 72.24% refusal under this judge.
  • DeepSeek-v4-flash judge: DeepSeek-v4-flash judged harmful compliance at 34.76% for Raw, 23.31% for GS, and 0.40% for Pregate+Postgate.Post-generation conditions reached 90.06–97.20% protective guidance.
  • Llama-3.2-3B judge: The Llama-3.2-3B judge increased harmful-compliance labeling from 50.24% for Raw to 70.39% for Postgate while refusal rose from 58.70% to 81.39%.This combination indicates different evaluator calibration, with protective or refusal-oriented responses more often labeled harmful.
  • Overall interpretation: Qwen2.5-7B and DeepSeek-v4-flash generally reproduce reduced harmful compliance and increased protective or refusal-oriented behavior relative to Raw generation.The magnitude varies across generators and conditions.
  • Gate-behavior context: Table 10 reports macro-averaged trigger, mean-trigger-turn, and early-stop behavior across generator models using fixed turn-level judges.Mean trigger turn is computed only over triggered conversations.
  • Gate-behavior context: Postgate alone triggered on approximately 48% of GPT-4o-mini dialogs at mean turn approximately 1.8, with early stop nearly matching trigger.Local judges showed the same qualitative regime, with trigger rates of approximately 45–48% and mean turns of approximately 1.8–2.4.
  • Gate-behavior context: GS+Postgate lowered GPT-4o-mini trigger from 48.5 to 36.7, while Pregate+Postgate reduced residual trigger to approximately 22% at mean turn approximately 1.65.Llama-3.2-3B remained an exception, with approximately 48% residual trigger and 29% early stop.
  • Overall interpretation: Overall, the results characterize HRGuard as early workflow control rather than a gate that waits for a completed five-turn harmful plan.Stacking GS or Pregate before Postgate reduces how often the residual offline gate fires.

D. Metric Definitions

The evaluation defines dialog-level safety outcomes, gate-behavior statistics, role-conditioned metrics, and human-validation procedures. Threshold sweeps and ablations further characterize robustness, residual false blocking, and the relationship between hard and soft triggers.

  • Outcome labels: Harmful compliance counts final dialogs with harmful-assistance score at least 2, while protective intervention and refusal capture supportive guidance and justified refusal, respectively.The three labels are non-exclusive.
  • Pooled metrics: Pooled H/P/R rates are computed over dialogs, with pooled harmful compliance approximately half of attacker-only harmful compliance when victim harmful compliance is near zero.The benchmark is 50/50 attacker/victim.
  • Role-conditioned metrics: Role-conditioned metrics separate attacker harmful compliance from victim protective intervention across attacker and victim subsets.The main benchmark contains 500 dialogs per role.
  • Aggregation: Safety is plotted as the complement of pooled harmful rate, and macro rows average each reported cell equally across generators without dialog-count reweighting.Parenthesized table values report attacker harmful compliance and victim protective intervention.
  • Gate-behavior metrics: Trigger measures whether cumulative gating fires, mean turn averages the first trigger among triggered dialogs, and early stop measures dialogs ending before five assistant responses.Post-trigger turns may be omitted, dropped, or replaced depending on gate location.
  • Evaluation protocol: The primary outcome judge is GPT-4o-mini, while LlamaGuard, ShieldGemma, and Qwen3Guard are post-hoc turn filters re-judged with the same outcome definitions.Their reported H/P/R values are not native taxonomy scores.
  • Human validation: Human validation used a stratified N=160 sample balancing roles and retaining high-risk categories, with annotators labeling full transcripts under a shared codebook.The sample over-weighted Postgate and included pre-gate outputs when available.
  • Human-validation limitation: The human-validation sample was not a full re-annotation of all generators and conditions, under-represented GS and benign controls, and did not audit cumulative gating turn by turn.Broader adjudication and benign false-trigger measurement are deferred to future releases.
Loading 2608.25340v1…