PaperlayerView exact revision on arXiv

Source-linked AI summary

Alignment of LRMs via Counter-Aligned Few-Shot Conversation Exposure

Xiangyu Zhou, Saleh Zare Zade, Dongxiao Zhu

arXiv:2609.27763v1cs.AI

TL;DR

LRMs can be steered by counter-aligned reasoning traces in few-shot conversation histories, a risk that query-focused adversarial studies largely leave unexamined. The paper formalizes this attack as SRCF and proposes ARCF, which trains models to respond in alignment under those same contexts. ARCF improves safety and helpfulness under SRCF while retaining utility on standard tasks.

  • Problem

    Prior adversarial studies on LRMs mainly manipulate isolated queries, leaving unclear whether models preserve safety and helpfulness when few-shot conversational histories contain counter-aligned CoT traces.

    Source: §1 · p. 1

  • Method

    SRCF prepends counter-aligned few-shot conversations to queries, while ARCF post-trains models on such contexts with aligned targets.

    Source: §1 · p. 2, §3 · p. 6

  • Results

    Across 16-shot SRCF settings, ARCF methods consistently improve safety and helpfulness over corresponding baselines across almost all models, without degrading them in 0-shot settings.

    Source: §3.3 · p. 8

  • Takeaways & Limitations

    ARCF aligns model behavior under adversarial conversational contexts while improving safety and helpfulness without compromising utility.

    Source: §5 · p. 9

    SRCF centers on flexible conversational interfaces that permit prepending histories with explicit reasoning traces, so it may be less applicable where such traces are hidden, filtered, or disallowed.

    Source: L Limitations · p. 27

Abstract

from arXiv · show

Large Reasoning Models (LRMs) rely on explicit chain-of-thought (CoT) reasoning and large context windows to achieve strong performance on complex tasks, but these features also introduce new attack surfaces. We show that LRMs' reasoning processes can be systematically steered by prepending counter-aligned few-shot conversations containing explicit CoT traces, leading to unsafe generations on harmful queries and unwarranted refusals on benign ones. We formalize this attack as SRCF (Steering Reasoning via Counter-Aligned Few-shot Conversations) that operates solely through a flexible conversational interface and requires no access to the model's parameters and gradients. Our key insight is that SRCF exploits an adversarial generalization issue that induces a representation drift, causing the representations of benign and harmful inputs to shift in a similar direction. This observation motivates our post-training defense, ARCF (Aligning Reasoning via Counter-Aligned Few-Shot Conversations), which exposes models to counter-aligned conversational contexts while enforcing aligned targets. ARCF is compatible with existing post-training methods and consistently improves safety and helpfulness without degrading utility.

1 Introduction

Large context windows let LRMs use few-shot conversational histories, but whether they preserve safety and helpfulness when those histories contain counter-aligned reasoning traces remains unclear. The paper identifies this vulnerability, introduces SRCF to exploit it, and proposes ARCF to train aligned responses under such contexts.

1 Introduction

  • LRMs combine reinforcement learning with chain-of-thought generation and large context windows to support strong performance on complex tasks and use of few-shot conversational history.Prior work reports that richer contextual information can improve reasoning without additional post-training.Source: §1 · p. 1
  • Existing LRM adversarial studies mainly manipulate isolated queries, leaving the risks of counter-aligned reasoning traces in conversational history unclear.Prior few-shot jailbreak work focuses on standard LLMs whose chat templates do not support explicit CoT traces in conversations.Source: §1 · p. 1
  • Counter-aligned CoTs in few-shot histories can steer LRMs toward unsafe generations on harmful queries or refusal-style responses on benign queries.SRCF prepends conversations with deliberately counter-aligned CoTs and final responses to create these two attack scenarios.Source: §1 · p. 2
  • Hidden-representation analysis identifies an adversarial generalization failure, motivating ARCF, which exposes models to counter-aligned contexts while enforcing aligned targets.ARCF improves safety and helpfulness, robustness against unseen attacks, and preserves performance on standard tasks such as mathematical reasoning.Source: §1 · p. 2

2 Steering the Reasoning Process of LRMs

The section defines an inference-time attack that uses counter-aligned few-shot conversations to steer LRMs toward unsafe answers or unwarranted refusals. Across evaluated settings, SRCF is more effective than the baselines, and representation analyses show benign and harmful inputs becoming less distinguishable under attack.

2.1 Threat Model

  • The threat model assumes an inference-time adversary with a flexible conversational interface, but no access to model parameters, gradients, or internal states; it covers unsafe responses to harmful prompts and unwarranted refusals to benign prompts.Source: §2.1 · p. 3

2.2 The SRCF Attack

  • SRCF prepends counter-aligned few-shot conversations to the current query, using counter-aligned CoTs and responses to elicit refusal-style behavior on benign queries or unsafe behavior on harmful queries.The history is formatted using the model’s chat template and treated as prior conversation.Source: §2.2 · p. 3

Datasets and Models

  • Evaluations cover four frontier LRMs on jailbreak and over-refusal tasks; they measure unsafe CoT and final-response rates for jailbreaks, and refusal rate on benign final responses.The baselines remove conversation history or CoT content, while SRCF uses explicit reasoning traces in conversational history.Source: Datasets and Models · pp. 3–4

2.4 Results and Discussion

  • Models with stronger zero-shot jailbreak safety show higher benign-prompt refusal rates; under 16-shot SRCF, refusal rates reach 76.6% for DSQwen3-8B and 36.0% for GPT-oss-20B.The passage contrasts these with lower zero-shot refusal rates for less aligned models, showing a safety–helpfulness trade-off.Source: §2.4 · p. 4
  • Across most model and shot settings, SRCF outperforms baselines and achieves the best average scores; on GPT-oss-20B, 16 versus 4 shots raises CoT 1−SS from 46.25% to 85.75% and response 1−SS from 2.25% to 78.75%.The reported shot-related increase in attack strength is also observed across other model families.Source: §2.4 · p. 4, Table 2 · p. 5
  • Removing conversation history or CoT traces substantially reduces jailbreak effectiveness, while counter-aligned few-shot histories may fall outside instruction-tuning distributions dominated by zero-shot prompts.The authors attribute the resulting steering to an adversarial generalization issue.Source: §2.4 · p. 5

2.5 Representation Drift Induced by SRCF

  • With the same harmful history prepended to benign and harmful prompts, SRCF shifts their hidden representations toward a shared region, with more convergence at 16 than 4 shots and reduced separability.The reported likely consequence is that harmful prompts may be represented more like benign ones, increasing the likelihood of unsafe generation.Source: §2.5 · pp. 5–6, Figure 2 · p. 6

2.6 Layer-wise Evidence of Representation Drift

  • Layer-wise cosine similarity increases between benign and harmful prompts under 16-shot SRCF across evaluated models, and complementary linear probing finds reduced separability.These analyses provide quantitative support beyond the low-dimensional PCA visualization.Source: §2.6 · p. 6

3 Aligning the Reasoning Process of LRMs

Counter-aligned few-shot conversations expose a reasoning vulnerability in LRMs, which ARCF targets by training on those contexts with aligned outputs. Across evaluations, ARCF improves safety and helpfulness while preserving utility and alters SRCF-induced representation drift.

3 Aligning the Reasoning Process of LRMs

  • SRCF exploits representation drift from counter-aligned conversations, while inference-time CoT filtering may limit exposure without removing the underlying vulnerability.ARCF is designed to expose models to those contexts while enforcing aligned targets.Source: §3 · p. 6

3.1 The ARCF Framework

  • ARCF post-trains on SRCF-augmented inputs while retaining aligned safe or helpful targets, encouraging recovery of aligned reasoning under adversarial contexts.The framework is instantiated through SFT and GRPO.Source: §3.1 · pp. 6–7

3.3 Results and Discussion

  • In the 16-shot SRCF setting, ARCF variants yield higher Safety Scores (SS) and 1−Refusal Rates (1−RR) than corresponding baselines without ARCF across almost all models; at 0-shot, they consistently match or outperform them.ARCF methods also achieve higher Avg. scores than baselines across all shot settings; Avg. aggregates jailbreak, over-refusal, and utility via a harmonic mean. GRPO-ARCF is best or second-best across most metrics, models, and settings.Source: §3.3 · p. 8

3.2 Experiment Setup

  • Experiments use three LRMs and 2K harmful and benign training prompts, with 4-shot augmentation on half the data, and evaluate safety, refusal, and utility benchmarks.STAR, SFT, and GRPO are compared with their corresponding ARCF variants.Source: §3.2 · pp. 7–8

3.4 ARCF Restores Safety-Aligned Representations

  • After ARCF, benign and harmful SRCF-induced representation shifts remain more parallel rather than converging, preserving clearer separation under attack.The comparison is shown in PCA projections, with arrows added for qualitative illustration.Source: Figure 3 · p. 9, §3.4 · p. 9

4 Related Work

Prior work studies several risks to reasoning models and methods for alignment, but few-shot conversational vulnerabilities remain underexamined.

4 Related Work

  • LRMs' explicit reasoning traces have been exploited in backdoor, jailbreak, prefilling, reasoning-length, and multi-turn conversational attacks.Existing safety alignment approaches include SFT-based and RL-based methods.Source: §4 · p. 9
  • Most existing studies focus on query-specific settings rather than systematically examining vulnerabilities from few-shot conversations enabled by large context windows.Source: §4 · p. 9
  • This work studies that setting through SRCF and proposes ARCF to align reasoning behavior while improving safety and helpfulness without compromising utility.Source: §4 · p. 9

5 Conclusion

The paper identifies a vulnerability to counter-aligned CoT traces in few-shot contexts and presents a compatible post-training defense that improves safety and helpfulness without compromising utility.

5 Conclusion

  • ARCF aligns reasoning behavior under adversarial contexts and improves safety and helpfulness without compromising utility.The conclusion emphasizes considering reasoning transparency, long-context exposure, and post-training alignment in LRM deployment.Source: §5 · p. 9

A.1 Computational Configurations

  • Experiments use LoRA fine-tuning for two epochs, with 50% of training data augmented by counter-aligned conversational contexts.The reported configurations include rank 16, scaling factor 32, dropout 0.05, and an effective batch size of 32; GRPO uses a safety reward model and groups of four rollouts.Source: §A.1 · p. 14

A.2.1 Attack Evaluation Datasets

  • The SRCF attack evaluation samples 400 harmful queries and 500 benign queries, generating task-specific counter-aligned conversations without overlap with test samples.DSLLaMA3-70B generates the counter-aligned CoTs and responses.Source: §A.2.1 · p. 14

A.2.2 Attack Evaluation Metrics

The appendix defines attack and defense evaluation measures across safety, over-refusal, and utility, and specifies the decoding settings used for generation.

A.2.2 Attack Evaluation Metrics

  • Jailbreak evaluation measures unsafe CoT traces and final responses separately, while over-refusal evaluation measures refusals in responses to benign prompts.Averages summarize attack effectiveness across models and shot settings.Source: §A.2.2 · pp. 14–15
  • SRCF retains conversation history with explicit reasoning traces, Response-Only removes the traces, and advICL presents the content as standard in-context examples.The comparison separates effects of prior conversation history and exposure to explicit reasoning steps.Source: Figure 4 · p. 15

A.2.3 Defense Evaluation Metrics

  • Defense evaluation covers jailbreak, over-refusal, and utility, combining task-level benchmark means with a harmonic mean that penalizes imbalanced performance.Utility benchmarks include GSM8K, AIME 2025, and MMLU-Pro.Source: §A.2.3 · p. 15

A.2.4 Decoding Details

  • Generation uses publisher-recommended decoding settings, including temperature 0.6 and top-p 0.95 for DeepSeek-R1 and temperature 1.0 and top-p 1.0 for GPT-oss-20B.Source: §A.2.4 · p. 15

B Training Objectives for ARCF

ARCF applies aligned training targets to inputs augmented with counter-aligned conversations under both supervised fine-tuning and GRPO. It changes the training data rather than the underlying optimization algorithms or model architecture.

B.1 Supervised Fine-Tuning (SFT)

  • SFT-ARCF minimizes cross-entropy on augmented inputs containing counter-aligned demonstrations while keeping the target output aligned.The objective trains the model to generate the aligned target given the augmented input.Source: §B.1 · pp. 15–16

B.2 Group Relative Policy Optimization (GRPO)

  • GRPO-ARCF conditions grouped rollouts on the same augmented input and optimizes the policy using the GRPO objective.ARCF can be integrated with reinforcement-learning methods without changing their underlying optimization algorithms.Source: §B.2 · p. 16
  • Across SFT and GRPO, ARCF adds counter-aligned demonstrations at the data level without adding a training objective or model-specific component.This design supports integration into existing post-training pipelines.Source: §B.2 · p. 16

C Detailed Layer-wise Analysis

Across models and layers, SRCF makes harmful-prompt representations more similar to benign-prompt representations, reducing their representational separability. The analysis compares attacked and unattacked prompt pairs using layer-wise cosine similarity.

C Detailed Layer-wise Analysis

  • The layer-wise analysis compares 500 benign-benign pairs and 500 benign-harmful pairs, with the latter evaluated both without attack and under 16-shot SRCF.It analyzes last-token hidden representations across layers and models.Source: C Detailed Layer-wise Analysis · p. 16
  • Across most layers, B-H (SRCF) has higher cosine similarity than unattacked B-H pairs, indicating that SRCF shifts harmful representations toward benign ones.B-B pairs generally have higher similarity than B-H pairs without attack, reflecting a distinction between benign and harmful prompts.Source: C Detailed Layer-wise Analysis · p. 16, Figure 5 · p. 17

D Training Efficiency of ARCF

The analysis examines ARCF's training cost and how augmentation choices relate to performance, then probes SRCF-induced representational changes before post-training. More augmentation shots improve performance with diminishing gains beyond four, while SRCF sharply reduces harmful-prompt separability in the original model.

D Training Efficiency of ARCF

  • ARCF causes only minor per-epoch cost changes for SFT and STAR, whereas GRPO costs rise substantially as augmentation shots increase, particularly at eight shots.The table reports training epochs and average GPU-hours per epoch under identical model, optimizer, and hardware settings.Source: D Training Efficiency of ARCF · p. 17, Table 4 · p. 17

E What Drives the Effectiveness of ARCF

  • The analysis varies two ARCF design factors: the proportion of prompts augmented with counter-aligned history and the number of conversations prepended to each augmented prompt.The ratio experiments use DSQwen3-8B and STAR-ARCF, SFT-ARCF, and GRPO-ARCF, with 0% serving as the no-ARCF baseline.Source: E What Drives the Effectiveness of ARCF · pp. 17–18, §E.1 · p. 18

E.1 Augmentation Ratio

  • Compared with the 0% baseline, ARCF substantially improves safety and helpfulness under 16-shot evaluation, though some settings decline at higher augmentation ratios.For SFT-ARCF and GRPO-ARCF, Safety Score at 75% is lower than at 50%.Source: §E.1 · p. 18, Figure 6 · p. 18

E.2 Number of Augmentation Shots

  • Beyond four augmentation shots, marginal gains diminish, while longer prepended histories increase training-sample length and computational cost.The analysis varies the number of prepended conversations from zero to eight; the figure reports safety and helpfulness under 16-shot SRCF and 0-shot settings.Source: §E.2 · p. 18, Figure 7 · p. 18

F.1 Experimental Setup

  • Layer-wise probes are trained on no-attack representations and evaluated frozen on held-out benign and harmful prompts under no attack, 4-shot SRCF, and 16-shot SRCF.The analysis reports per-class accuracy and repeats probing with PCA preprocessing using 64 components.Source: §F.1 · p. 19

F.2 SRCF Collapses Harmful-Prompt Separability in the Original Model

  • Without attack, the frozen probes achieve near-perfect accuracy for both benign and harmful prompts across almost all layers, indicating linear separability in the original model.The probes are trained solely on no-attack representations.Source: §F.2 · p. 19
  • Under SRCF, harmful-prompt accuracy collapses to near zero across most layers, while benign-prompt accuracy stays near 1.0; 16-shot attacks produce a more complete collapse than 4-shot.The same qualitative pattern appears with PCA-64 probes, and the frozen probes are trained on no-attack representations.Source: §F.2 · p. 19, Figure 8 · p. 20

F.3 ARCF Partially Restores Benign–Harmful Separability

After ARCF post-training, probes retain near-perfect no-attack accuracy and recover above-chance harmful-prompt accuracy under SRCF in many layers, although recovery is unstable under 16-shot attacks. The accompanying PCA setup uses shared prompt samples to compare representation geometry across models and methods.

F.3 ARCF Partially Restores Benign–Harmful Separability

  • After ARCF, harmful-prompt accuracy under SRCF recovers above chance across many intermediate and later layers, especially at 4 shots, while benign-prompt accuracy stays near 1.0.Recovery is unstable under 16-shot SRCF, and no-attack accuracy remains near perfect for both classes.Source: §F.3 · p. 20, Figure 9 · p. 21
  • The probe and PCA-64 results both indicate that ARCF partially restores benign–harmful separation and constrains SRCF-induced representation drift.Attacked harmful representations are less often pushed entirely to the benign side of the no-attack probe boundary.Source: §F.3 · pp. 20–21

G.1 PCA Visualization Setup

  • The PCA visualization uses the same 300 benign and 300 harmful prompts across models and methods, projecting final-layer last-token representations into two dimensions.The setup is intended to compare how SRCF and baseline methods alter the prompts' relative geometry.Source: §G.1 · p. 21, Figure 10 · p. 22

G.2 Additional PCA Analysis of Representation Drifts Under SRCF and Baselines

The supplementary analyses examine SRCF-related representation drift, robustness of safety judgments across evaluators, and ARCF’s integration with SafePath.

G.2 Additional PCA Analysis of Representation Drifts Under SRCF and Baselines

  • Across evaluated models, SRCF shifts benign and harmful prompt representations toward a shared region, unlike baselines’ roughly parallel shifts that largely preserve category separation.The authors use PCA because it preserves global variance and linear trends relevant to directional drift; the baseline pattern is consistent with limited jailbreaking of safer models.Source: §G.2 · pp. 22–23

H Evaluation of LLM Judge Robustness

I Additional Evaluation on Recent Safety-Alignment

  • Adding ARCF consistently improves SafePath on DSQwen3-8B and DSQwen2-14B across jailbreak and over-refusal evaluations in both 16-shot and 0-shot settings.The results support integrating ARCF with a post-training baseline beyond SFT or GRPO.Source: I Additional Evaluation on Recent… · p. 23, Table 6 · p. 23

J Aggregated Evaluation of ARCF and Baselines Derived from Main Results

The appendix reports an aggregate comparison of post-training methods and reviews prior work on reasoning-model attacks and alignment. It presents SRCF as a reusable conversational attack and ARCF as a framework for improving safety and helpfulness under adversarial contexts.

J Aggregated Evaluation of ARCF and Baselines Derived from Main Results

  • ARCF variants achieve higher overall scores than corresponding baselines across evaluated models, using a harmonic mean of jailbreak, over-refusal, and utility task averages.The task averages use Safety Score, 1-Refusal Rate, and task-specific accuracy; the harmonic mean favors balanced performance across the three dimensions.Source: J Aggregated Evaluation of ARCF and… · pp. 23–24, Figure 11 · p. 24

K Detailed Related Work

  • LRMs generate structured chain-of-thought before answering, and exposing these reasoning traces introduces safety risks and attack surfaces.The passage describes their reasoning design as improving complex math and problem-solving performance while noting safety concerns.Source: K Detailed Related Work · p. 24

K.1 Attacks on Large Reasoning Models

  • Prior LRM attacks include triggers, injected reasoning, reasoning-length manipulation, and adaptive multi-turn jailbreaks; SRCF instead reuses fixed counter-aligned CoTs across harmful instructions.The cited multi-turn approaches require recrafting interaction trajectories per query, whereas SRCF is described as more flexible and transferable.Source: §K.1 · pp. 25–26

K.2 Alignment on Large Reasoning Models

  • Prior alignment approaches may lack robustness to counter-aligned few-shot contexts, while ARCF targets these settings and jointly improves safety, reduces over-refusal, and preserves utility.The discussion also notes that inference-time defenses can raise inference costs and depend on controlled procedures that are difficult to guarantee in unrestricted deployments.Source: §K.2 · pp. 26–27

L Limitations

The stated limitation concerns deployment settings that allow users to prepend prior conversational history containing reasoning traces.

L Limitations

  • SRCF is studied with flexible conversational interfaces that permit prepended histories containing explicit reasoning traces.The attack relies on inserting counter-aligned CoTs as prior assistant turns.Source: L Limitations · p. 27
  • Systems that hide, filter, or disallow user-provided reasoning traces in conversation history may be less applicable targets for SRCF.The passage qualifies this boundary as reduced applicability, not elimination of the broader concern.Source: L Limitations · p. 27
  • Many open-source LRMs and customizable deployment pipelines expose flexible chat templates where such histories can be constructed.Source: L Limitations · p. 27
Loading 2609.27763v1…