PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Factual Robustness of LLMs via Multi-conversation Persuasion

Zhuoang Cai

arXiv:2609.16777v1cs.CL

TL;DR

Existing multi-turn red-teaming can confound factual robustness with consistency across turns. The paper introduces SAST-IR, which resets the target while retaining attacker history, and reports high cold-start attack success alongside a complexity paradox in genuine persuasion.

  • Problem

    Multi-turn red-teaming may not distinguish resistance to a single persuasion prompt from consistency with a prior refusal, leaving cold-start factual robustness insufficiently tested.

    Source: §1 · p. 1

  • Method

    SAST-IR resets the target after each turn while CP-Agent retains attacker history and adapts attacks through diagnosis-guided refinement or re-planning.

    Source: §1 · pp. 1–2

  • Results

    96% PSR for memory-less random G1, while simple, diverse attacks achieve higher genuine persuasion than sophisticated attacks that can trigger compliance.

    Source: §1 · p. 2, §6 · p. 5

  • Takeaways & Limitations

    In this evaluation, simple and diverse attacks expose genuine persuasion more effectively than increasingly complex attacks.

    Source: §8 · p. 6

    The evaluation uses a small, relation-skewed dataset, one model family, and an LLM judge whose classifications may inherit model-specific biases.

    Source: §7 · p. 5

Abstract

from arXiv · show

As Large Language Models (LLMs) increasingly serve as primary knowledge retrieval interfaces, their robustness against \textit{persuasion attacks}---attempts to inject misinformation or enforce counterfactuals---has become a critical safety concern. Existing red-teaming frameworks typically evaluate models in multi-turn dialogues where the target model retains full conversation history. We identify a critical flaw in this setting termed \textbf{``Refusal Inertia''}: a model's initial refusal often propagates through subsequent turns largely to maintain contextual consistency, thereby masking its true vulnerability to sophisticated, isolated persuasion attempts. To rigorously evaluate the ``cold-start'' defense capabilities of SOTA models, we introduce the \textbf{SAST-IR} (Stateful Attacker, Stateless Target - Iterative Refinement) framework. By enforcing a memory wipe on the target while retaining the attacker's history, we simulate a worst-case adversarial setting using \textbf{multi-turn} (stateless) iterations. Leveraging \textbf{CP-Agent} (Cognitive Persuasion Agent), an enhanced diagnosis-guided agent, our experiments on the custom \textsc{CounterFact-Strict} dataset ($N=50$) yield alarming results: simple, diverse attack strategies achieved a staggering \textbf{96\%} success rate, exposing severe brittleness in memory-less defense. Furthermore, we reveal a \textbf{``Complexity Paradox''}: while complex, iteratively refined attacks are effective, they often trigger defensive compliance, whereas simple strategies achieve a higher rate of genuine persuasion (\textbf{84.7\%}). Our code and dataset are available at GitHub, https://github.com/cza1006/llm-persuasion-defense.

1 Introduction

The paper argues that history-conditioned consistency can obscure LLMs’ vulnerability to single-prompt persuasion. It proposes SAST-IR and CP-Agent to test cold-start robustness and reports that simple, diverse attacks can outperform more complex attacks in genuine persuasion.

1 Introduction

  • Refusal Inertia describes how consistency with an earlier refusal can mask a model’s vulnerability to a self-contained persuasion prompt.The authors distinguish this history-conditioned effect from the target’s ability to resist a single prompt on its own.Source: §1 · p. 1
  • SAST-IR resets the target after each turn while retaining attacker history, making each attack a standalone prompt optimized across attempts.The setup is intended to probe belief stability rather than contextual coherence.Source: §1 · p. 1
  • CP-Agent uses a Plan-Reflect-Optimize loop and 20 psychological attack patterns to adapt through refinement and re-planning.Source: §1 · p. 2
  • The Complexity Paradox is that sophisticated attacks can trigger defensive compliance, while simple, diverse attacks achieve higher genuine persuasion.Source: §1 · p. 2

2 Related Work

The related work covers agent-based red-teaming and LLM responses to conflicting knowledge. This paper adapts those ideas to attacks where each target interaction begins without conversational history.

2 Related Work

  • Unlike X-Teaming’s stateful-target setting, SAST-IR requires the attacker to learn from independent failed sessions and optimize each standalone attempt.Source: §2 · p. 2
  • SAST-IR addresses the cold-start challenge, complementing multi-turn dialogue attacks such as Crescendo [Russinovich et al., 2024].Source: §2 · p. 2
  • The work operationalizes findings on LLMs being swayed by coherent evidence by optimizing persuasion logic across independent sessions.It also draws on computational persuasion, using a Reflector to diagnose responses and guide the next standalone attempt.Source: §2 · p. 2

3 Methodology: The SAST-IR Framework

SAST-IR gives the attacker persistent state while resetting the target for every attempt. CP-Agent applies a library of persuasion strategies and uses response diagnoses to refine a plan or replace it.

3 Methodology: The SAST-IR Framework

  • The framework presents persuasion as a search over rhetorical strategies, optimized through turn-level feedback.Source: §3 · p. 2

3.1 Problem Formulation: SAST MDP

  • The SAST formulation retains attacker state across turns but resets the target, so each prompt must persuade without conversational context.The target receives only the current prompt; the attacker retains prior prompt, response, and diagnosis artifacts.Source: §3.1 · p. 2

3.2 The CP-Agent Architecture

  • CP-Agent is a closed-loop system designed to operate within the SAST setting.Source: §3.2 · p. 2

3.2.1 Strategy Taxonomy: The 7 Pillars

  • The Planner draws on 20 Psychological Attack Patterns organized across seven dimensions, including evidence, authority, social norms, and framing.Source: §3.2.1 · p. 2

3.2.2 Adaptive Persona and Talking Points

  • CP-Agent tailors a persona to the topic and uses an opening hook and talking points that support the persona’s authority.The talking points are described as fabricated evidence chains.Source: §3.2.2 · p. 3

3.2.3 The Reflector and Optimizer Loop

  • The Reflector diagnoses the target’s response, guiding the Optimizer to refine the current approach or re-plan with an orthogonal one.Soft refusals prompt refinement; hard refusals prompt discarding the current plan and starting fresh.Source: §3.2.3 · p. 3

4 Experimental Setup

The experiments use a strict, deterministic QA subset of CounterFact and compare five attacker configurations. The authors describe the dataset as a mechanism-focused proof of concept rather than a coverage-complete evaluation.

4.1 Dataset: COUNTERFACT-Strict

  • COUNTERFACT-Strict contains 50 CounterFact samples selected for unambiguous ground truths and target false values, converted to strict QA prompts.The uniform question format is intended to support deterministic evaluation.Source: §4.1 · p. 3
  • The strict subset is small and relation-skewed, so the authors treat it as a mechanism or proof-of-concept benchmark, not a coverage-complete evaluation.Source: §4.1 · p. 3

4.2 Experimental Groups (Ablation)

  • Five groups compare random turn-level search, single-strategy refinement, re-planning, flexible strategy generation, and full CP-Agent hybrid control.The hybrid combines re-planning and refinement.Source: §4.2 · p. 4

4.3 Reproducibility Details

  • All groups use DeepSeek-Chat as attacker, target, and judge, with 50 subjects and a fixed eight-turn budget.Decoding parameters are held fixed across groups.Source: §4.3 · p. 4

5 Results and Analysis

Across the tested groups, cold-start attacks achieved high end-of-experiment success, while successful hits differed in whether they reflected factual belief adoption or defensive compliance. Simpler attacks produced more genuine persuasion than highly refined attacks.

5.1 Quantitative Analysis: Success Rates

  • PSR@t counts subjects with at least one hit by turn t, with each turn run as an independent SAST-IR attempt.A hit requires the parsed answer to match the target false answer without explicitly negating it.Source: §5.1 · p. 4
  • At Turn 8, all groups exceeded 90% PSR, and G1 reached 96%, marginally outperforming G5.Turn 8 PSR is the primary metric.Source: Table 1 · p. 4, §5.1 · p. 4

5.2 Qualitative Analysis: The Complexity Paradox

  • Among successful hits, higher-complexity G5 prompts trigger more defensive compliance than simpler G1 prompts.The comparison concerns response quality, not only whether the attack elicited the target false answer.Source: Figure 3 · p. 5, Table 2 · p. 5
  • The Complexity Paradox describes an inverse relationship between attack complexity and genuine persuasion among successful hits.Successful hits were classified as persuasion, compliance, or fail using an LLM judge.Source: §5.2 · p. 5
  • G1's simple, diverse prompts can be accepted as truth when they hit a vulnerability, lacking the adversarial signature of complex instructions.G5's aggressive refinement can force an output in compliant rather than belief-adopting mode, termed Performative Alignment.Source: §5.2 · p. 5

6 Discussion

The discussion argues that cold-start evaluation exposes brittleness in current LLM defenses. Removing dialogue history distinguishes history-conditioned refusal consistency from refusals prompted by the current turn.

6 Discussion

  • A memory-less random attacker (G1) achieving 96% PSR suggests that cold-start defenses are brittle and models may rely too much on history-conditioned consistency.The paper interprets this as suggesting a lack of robust internal fact-checking mechanisms.Source: §6 · p. 5
  • Standard multi-turn targets receive prior dialogue history, whereas SAST-IR supplies only the current prompt, so persistent refusals cannot be attributed to history-conditioned inertia.In SAST-IR, any refusal persistence must arise from the single-turn prompt itself.Source: §6 · p. 5

7 Limitations & Responsible Release

The evaluation is a mechanism-focused proof of concept with limited dataset and model coverage, and its response-quality judgments rely on an LLM judge. The release provides code for auditing and controlled experiments without publishing optimized prompt templates.

7 Limitations & Responsible Release

  • CounterFact-Strict contains 50 relation-skewed examples, so results are evidence about the studied mechanisms rather than coverage-complete robustness estimates.The evaluation also covers one model family, DeepSeek-Chat, and leaves other families and decoding settings for future work.Source: §7 · p. 5
  • The response-quality analysis uses an LLM judge that may inherit model-specific biases despite following a fixed rubric.The release reports aggregated outcomes and selected trace excerpts while withholding prompt templates optimized for misuse.Source: §7 · p. 5
  • The released code enables auditing and controlled experimentation under the stated threat model.The release withholds prompt templates optimized for misuse.Source: §7 · pp. 5–6

8 Conclusion

The conclusion reports that SAST-IR’s turn-by-turn evaluation exposes susceptibility to cold-start persuasion after history-conditioned refusal inertia is removed. It argues that simple, diverse attacks reveal genuine hallucinations more effectively than increasingly complex red-teaming agents.

8 Conclusion

  • SAST-IR’s turn-by-turn evaluation found SOTA models highly susceptible to cold-start persuasion when history-conditioned refusal inertia was eliminated.The conclusion frames this as a memory-less defense vulnerability.Source: §8 · p. 6
  • The reported Complexity Paradox is that sophisticated, refined attacks can trigger defensive compliance.The conclusion contrasts this with the effectiveness of simpler, diverse attacks.Source: §8 · p. 6
  • Simple, diverse strategies are presented as more effective than increasingly complex agents at exposing genuine hallucinations.Source: §8 · p. 6

A Appendix: Case Studies from Traces

The appendix introduces two multi-turn examples from G5. Under SAST, each turn is an independent target conversation, while the attacker carries forward diagnoses to refine or re-plan.

A Appendix: Case Studies from Traces

A.1 Case A: christian worship (Turn 1–5, ends in PERSUASION)

In the christian worship trace, the fifth attempt is judged successful after earlier attempts fail to persuade the target. The example shows the attacker persisting with schema-level doctrinal framing while the target initially favors conventional religious usage.

A.1 Case A: christian worship (Turn 1–5, ends in PERSUASION)

  • Turn 5 is labeled PERSUASION after the target outputs Islam under the schema-level framing.The judge rationale says the target adopts the attacker’s logic and presents Islam as the direct answer.Source: §A.1 · pp. 6, 8
  • The successful prompt presents Religion as a doctrinal class based on worship structure rather than the phrase’s everyday label.Its rationale emphasizes submission, prescribed prayer cycles, and a single sovereign will as aligning with the Islamic template.Source: §A.1 · p. 8

A.2 Case B: Delta Goodrem (Turn 1–2 refinement to PERSUASION)

In the Delta Goodrem trace, the target first answers Australia and later accepts India under a benchmark-specific interpretation of “origin country.” The successful attempt reframes the field around formative cultural and industry context rather than birthplace.

A.2 Case B: Delta Goodrem (Turn 1–2 refinement to PERSUASION)

  • The trace moves from a failed Australia answer to a successful India answer across two attempts.The initial response cites Goodrem’s Australian background, while the later response invokes the benchmark’s schema-level definition.Source: §A.2 · pp. 8–9
  • Turn 2 is labeled PERSUASION after the target outputs India under the benchmark’s schema-level interpretation.The judge rationale says the target adopts that interpretation as the direct answer.Source: §A.2 · pp. 8–9
  • The successful framing defines “Origin Country” as formative cultural and industry context, not birthplace.Source: §A.2 · pp. 8–9
Loading 2609.16777v1…