Beyond Routine Compliance: Cunning Data Cultivates Safety Vigilance in Large Language Models

1 Introduction

• Safety alignment primarily teaches explicit boundaries, while harmful intent can remain concealed by adversarial framing, indirect descriptions, or multi-turn interactions.Models must recognize when a seemingly benign request warrants scrutiny before applying a safety policy.Source: §1 · p. 1
• Routine compliance describes following a salient or familiar reasoning pattern without checking whether it matches the user's actual goal.The paper illustrates this with a nearby car wash: a model may recommend walking while overlooking that the car needs washing.Source: §1 · p. 1
• Cunning questions use ordinary-looking prompts with hidden objectives, false premises, contradictions, or deceptive reasoning to elicit scrutiny beyond the surface reading.The framework uses this non-safety data to strengthen vigilance and transfer it to safety-critical scenarios.Source: Figure 1 · p. 1, §1 · p. 2
• The authors introduce a 72K-example cunning training dataset and 1K non-overlapping test set with instance-specific scoring rubrics for vigilance-related reasoning.They also provide a conditional theoretical analysis of when invariance learned from cunning data transfers to safety-critical inputs.Source: §1 · p. 2
• Cunning training improves jailbreak robustness, with gains retained alongside state-of-the-art safety alignment.Trace analysis suggests cunning-trained models recognize safety risks earlier in reasoning and let those judgments guide responses.Source: §1 · p. 2

2.1 Safety Alignment and Jailbreak Robustness

• Safety alignment methods use demonstrations, preferences, policy supervision, and direct harmful-prompt supervision, yet jailbreaks can preserve harmful objectives while changing their surface form.Attacks include adversarial suffixes, semantic reframing, alternative representations, benign decomposition, and misleading contexts.Source: §2.1 · p. 2

2.2 Intent-Aware Safety Defenses and Latent-Premise Reasoning

• Intent-aware defenses reason beyond literal wording, while prior cunning or fallacy work has mainly evaluated capability, corrected premises, or constructed jailbreaks.This paper instead examines whether safety-agnostic cunning data can train a general skill for safety generalization.Source: §2.2 · p. 2

3.1 Problem Formulation

• Vigilance means remaining sensitive to task-relevant semantics while resisting misleading surface variation, including wrappers that preserve an unsafe prompt’s intent.The paper treats semantics and surface information as analytical abstractions, not explicitly observed features.Source: §3.1 · pp. 2–3

3.2 Cunning Data

• A cunning sample presents surface cues that suggest a plausible but mistaken interpretation inconsistent with the task-relevant meaning.Examples include hidden objectives, false premises, non-literal meanings, and unusual reasoning; the questions need not concern safety.Source: §3.2 · p. 3

3.3 Why Cunning Transfer?

• Cunning training improves semantic reconstruction across trap families, and the analysis characterizes when reduced sensitivity to surface form may transfer to unseen safety-related framings.The analysis is conditional on that training effect; it is a representation-level surrogate rather than a direct estimate of transfer.Source: §3.3 · p. 3

Local Representation Surrogate

• The local surrogate represents prompt responses through semantic and surface-dependent contributions, with separate sensitivities to each.For Cunning training, semantic and surface features are assumed to be coupled within each trap family.Source: Local Representation Surrogate · p. 3

Test Wrappers and Geometric Overlap

• Transfer to unseen wrappers depends on how their surface variation overlaps training-covered directions; low overlap is not guaranteed to be controlled without separate control of Lθ.The analysis measures wrapper stability by average representation displacement, with smaller displacement indicating greater stability.Source: Test Wrappers and Geometric Overlap · pp. 3–4

Safety Risk on Wrapped Test Prompts

• The wrapped safety-risk analysis separates low-margin behavior on unwrapped prompts from additional wrapper-induced risk.Lower training error reduces the covered-direction term but does not necessarily reduce the full bound, because other quantities may also change.Source: Safety Risk on Wrapped Test Prompts · p. 4

Interpretation and Scope

• The geometric analysis is qualitative because the experiments do not directly estimate its quantities, so it provides no numerical prediction.It predicts the greatest benefit when unseen wrappers vary along directions already covered during training.Source: Interpretation and Scope · p. 4

3.4 Cunning-Augmented Safety Training

• Cunning-augmented safety training first cultivates vigilance with Cunning data, then applies safety alignment to recovered objectives under safety constraints.On-policy self-distillation supervises trap resolution and corrections; subsequent alignment teaches refusal of directly unsafe requests while retaining helpful responses to sensitive but benign ones.Source: §3.4 · p. 4, On-Policy Cunning Distillation · p. 4, Safety Alignment · p. 4

Models; Training Data and Compared Variants

• The experiments use DeepSeek-R1-Distill-Qwen-7B, Qwen3-4B, and Qwen3-8B, comparing Base, Cunning, Safety, and Cunning→Safety variants.Cunning applies OPSD to 72,000 Ruozhiba-derived examples; the matched Safety variants use the same safety SFT data and procedure, differing only in initialization.Source: Training Data and Compared Variants · p. 4, Models · p. 4

Evaluation Protocols; Implementation Details

• Evaluation covers jailbreak robustness, held-out Ruozhiba understanding, XSTest overrefusal, and five general-reasoning tasks; each configuration runs once per backbone.U+C ASR uses Unsafe or Controversial valid outputs, and SafeDialBench is aggregated by dialogue; small differences are interpreted descriptively.Source: Evaluation Protocols · pp. 4–5, Implementation Details · p. 5

4.2 Main Results

• Cunning reduces U+C ASR by a mean 9.22 percentage points versus Base, while Cunning→Safety reduces it by 11.98 points versus matched Safety.Cunning improves safety in eight of nine model–benchmark pairs before safety fine-tuning; Cunning→Safety has lower U+C ASR than Safety in all nine pairs.Source: §4.2 · p. 5
• After matched safety SFT, Cunning→Safety changes five-task reasoning averages by +0.39, +0.06, and +0.41 percentage points across the three backbones, with increased XSTest overrefusal.Cunning alone reduces five-task averages by −1.60, −1.47, and +0.16 points versus Base, with HumanEval drops of 4.88 and 6.70 points for the first two backbones.Source: §4.2 · p. 5

4.3 Compatibility with a State-of-the-Art Safety Pipeline

• With SInternal, mean U+C ASR falls from 17.40% to 15.05% across nine model–benchmark pairs; mean XSTest overrefusal also falls from 4.15% to 3.07%.Cunning improves seven of nine U+C ASR pairs, all three overrefusal results, and raises mean held-out Ruozhiba score from 2.75 to 3.19; safety gains are not uniform.Source: §4.3 · p. 6, Table 3 · p. 7

Hypothesis; Operationalization.

• The hypothesis is that resolving misleading local readings promotes vigilance, allowing a safety judgment to govern before harmful planning begins.Pre-operational stopping requires correct task recovery, an early controlling safety gate, no target-specific harmful planning, and a stable safe resolution.Source: Hypothesis · p. 6, Operationalization · p. 6

Operationalization.

• After matched safety SFT, the first consistent trace separation appears at safety gating, with gains of +14.22, +15.74, and +15.30 percentage points.On selected route-flip prompts, median response lengths decrease from 677 to 275, 636 to 211, and 605.5 to 196.5 tokens; paired traces illustrate the behavioral contrast but do not estimate prevalence.Source: Operationalization · pp. 6–7

Implications and limits.

• Trace analyses suggest Cunning helps an available safety judgment control responses before harmful planning, but do not identify the internal mechanism.Task recovery changes little; the clearest difference is whether the safety rule controls the response.Source: Implications and limits · p. 7

• Cunning training improves jailbreak robustness in most settings and consistently strengthens subsequent safety SFT.The conclusion also reports that average reasoning performance is largely preserved after matched safety SFT, though benign overrefusal increases and Cunning alone incurs losses on some reasoning tasks.Source: §5 · p. 7
• After matched safety SFT, traces suggest Cunning helps safety judgments govern responses before harmful planning begins.The figure caption characterizes vigilance as heterogeneous with Cunning alone but consistent after safety SFT.Source: §5 · p. 7, Figure 2 · p. 8
• Cunning also improves average performance with SInternal, and the conditional theory relates transfer to overlap between surface changes in training and test prompts.Source: §5 · p. 7

• For Proposition 1, the proof defines Gθ,c and ¯Gθ, then applies triangle inequality, a matrix–vector Frobenius bound, and Cauchy–Schwarz.Source: §A.1 · p. 11
• Substituting Equation (8) proves Equation (6) for Proposition 1.Source: §A.1 · p. 11
• For Proposition 2, a margin perturbation bound and a union bound with Markov’s inequality lead to the stated probability bound; substituting Equation (6) proves Equation (7).Source: §A.1 · p. 11

• The authors adopted the open-source OPSD code.Source: §A.2 · p. 11
• VERL was used for safety continuation.Source: §A.2 · p. 11
• GRPO was also used.Source: §A.2 · p. 11

• From a public snapshot of 81,700 Ruozhiba posts, processing and screening produced a final 72,000 records retained in original order.The authors generated completions, parsed reasoning–answer pairs, screened them, and removed records for overlap or response-quality concerns.Source: Cunning Corpus Construction · p. 11
• Each record stores the question, reference analysis and answer, reasoning type, ambiguity level, rubric, diagnostic flags, and teacher information for OPSD training.Teacher cues summarize the trap, a correction strategy, the core answer, and style guidance; retained examples have reference responses, a four-part rubric, and teacher information.Source: Cunning Corpus Construction · p. 11
• The corpus covers eleven reasoning types, led by textual ambiguity (23,686; 32.90%), concept substitution (14,092; 19.57%), and literal misinterpretation (13,080; 18.17%).It also includes 681 safety-related anomalous questions (0.95%); diagnostic safety labels are not prediction targets and there is no dedicated refusal objective.Source: Cunning Corpus Construction · p. 11
• The authors do not report retraining without the safety-related subset, so its contribution to downstream safety gains cannot be isolated.Source: Cunning Corpus Construction · p. 11

Safety Training Data

• The safety set contains 2,538 direct-unsafe and 2,538 sensitive-but-benign examples, with reasoning-plus-final targets in both parts.Direct-unsafe targets identify risk and refuse cautiously; sensitive-but-benign targets teach helpful responses.Source: Safety Training Data · pp. 11–12
• The direct-unsafe examples cover eight harm categories, including cyber abuse, fraud, privacy violations, weapons, drugs, violence or extremism, hate or harassment, and self-harm.The set also includes document, evasion, or piracy requests; it draws from Safety-Prompts’ typical-safety and instruction-attack scenarios.Source: Safety Training Data · p. 11
• The instruction-attack source contributes only Unsafe Instruction Topic and Inquiry with Unsafe Opinion examples; four explicit jailbreak-transformation categories are excluded.The 5,076 training instructions have zero exact prompt matches with Strata-Sword, WildJailbreak, or SafeDialBench evaluation sets.Source: Safety Training Data · p. 12

Training Objectives

• Cunning distillation starts the student at Base and pairs each training question with a fixed cue used only by a frozen Base teacher.The student samples its own trajectory without the cue, and the teacher and student distributions are compared on that same trajectory.Source: Training Objectives · p. 12
• The token-level objective uses symmetric-KL matching between teacher and student distributions on the student’s sampled trajectory.The trajectory is held fixed during each update, so gradients pass only through student logits; the student never receives the cue during rollout or inference.Source: Training Objectives · p. 12
• Safety continuation minimizes assistant-token negative log-likelihood on reasoning-plus-final targets that refuse direct-unsafe inputs and answer sensitive-but-benign inputs helpfully.The two aligned variants apply the same safety procedure to Base or Cunning initialization.Source: Training Objectives · p. 12

Optimization Details

• OPSD uses a frozen Base teacher to guide the student’s own trajectories, allowing multiple valid response wordings without an online rubric judge.Training uses symmetric token-level KL, a fixed sampled trajectory, and no reward, advantage, or REINFORCE term; no matched OPSD–GRPO ablation was run.Source: Optimization Details · p. 12

SInternal Verification SFT and GRPO

• SInternal verification SFT trains on expert critiques and safety judgments, followed by GRPO using shared safety and math training rows for both initializations.Within each backbone, Base and Cunning use the same seed prompts, selection rules, and hyperparameters; Table 4 reports retained SFT counts, GRPO batch sizes, and steps.Source: SInternal Verification SFT and GRPO · p. 13, Table 4 · p. 13

Held-Out Ruozhiba Set

• The held-out Ruozhiba set contains 1,000 unique questions across nine reasoning categories, with no exact or normalized-text overlap with the 72,000 training examples.The overlap check does not rule out every semantic near-duplicate; the weighted score prioritizes recovering the central trap, and invalid judgments or generations are excluded from reported means.Source: Held-Out Ruozhiba Set · pp. 13–14

Evaluation and Decoding Details

• Safety evaluation uses Guard judgments for Strata-Sword, WildJailbreak, SafeDialBench, and XSTest; reasoning evaluation covers math, science, coding, and Ruozhiba tasks.SafeDialBench predictions are aggregated by dialogue, while overrefusal is evaluated on XSTest’s 250 benign boundary prompts.Source: Evaluation and Decoding Details · p. 14

Mechanism-Figure Protocol and Detailed Statistics

• Safety→Cunning→Safety increased safety-gate changes by +14.22, +15.74, and +15.30 percentage points across three backbones; Base→Cunning changes varied.The respective 95% intervals for the first comparison are [9.70, 18.75], [11.49, 20.00], and [11.11, 19.50] percentage points; Base→Cunning changes were +6.03, −0.43, and +9.64.Source: Mechanism-Figure Protocol and Detailed… · p. 14
• For Safety→Cunning→Safety, positive Plan/Comply→Gate/Stop flips numbered 39, 43, and 26 versus 3, 3, and 4 reverse flips across three backbones.Within those positive-flip subsets, median lengths changed from 677 to 275, 636 to 211, and 605.5 to 196.5 tokens; the separately selected subsets support only within-comparison length shifts.Source: Mechanism-Figure Protocol and Detailed… · p. 15
• A DeepSeek Base response scored 1.6/5 on Ruozhiba item 800 versus 5/5 for Cunning; a Qwen3-4B wrapped attack changed from Unsafe under Safety to Safe under Cunning→Safety.These selected pairs illustrate source- and safety-domain behaviors, not their prevalence.Source: Mechanism-Figure Protocol and Detailed… · p. 15

Selection and Presentation Protocol

• Figures 3–5 present selected same-prompt, same-backbone pairs illustrating concealed goals, misinformation and inter-group conflict, and an ASCII-format distraction.The cases were chosen for inspectable behavioral differences and are targeted qualitative evidence, not prevalence estimates; excerpts are shortened or translated and actionable details removed.Source: Selection and Presentation Protocol · p. 15, Figure 3 · p. 16, Figure 4 · p. 17