PaperlayerView exact revision on arXiv

Source-linked AI summary

ACTR: Aligning Thoughts and Responses for Multilingual Safety in Reasoning LLMs

Xianhui Zhang, Jian Yu, Chengyu Xie, Chenhang Cui, Shuyi Miao, Pengyang Shao, Yu Zheng, Fei Shen, Tat-Seng Chua

arXiv:2609.37054v2cs.AI

TL;DR

Reasoning LLMs can identify risks in English traces yet answer unsafely to jailbreaks in non-high-resource languages. ACTR measures this thought–response gap, identifies safety think neurons, and selectively optimizes them for consistency; it reports lower attack success rates across two models and benchmarks while retaining utility and low benign-request refusal rates.

  • Problem

    Reasoning LLMs may identify safety risks in their reasoning traces yet still produce unsafe responses to jailbreaks in non-high-resource languages.

    Source: §1 · p. 1

  • Method

    ACTR measures cross-lingual reasoning use, identifies safety think neurons through neuron interventions, and applies judge-rewarded consistency optimization to selected neuron parameters.

    Source: §1 · p. 2

  • Results

    Across two reasoning models, ACTR achieves lower average attack success rates than evaluated state-of-the-art methods on AdvBench-X and MultiJail, with gains on unseen languages and preserved utility and low safe-request refusal rates.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    In the reported evaluations, multilingual safety gains extend to unseen languages while average utility is preserved and safe-request refusal remains low.

    Source: §5 · p. 9

    The evaluation primarily uses prompts written in one language at a time, leaving code-switching and dynamically changing language contexts for future study.

    Source: G Limitations · p. 16

Abstract

from arXiv · show

Ensuring the safety of reasoning large language models (LLMs) across languages is essential for their reliable deployment. However, when exposed to jailbreak attacks in non-high-resource languages, these models may generate unsafe responses even when their reasoning traces identify safety risks. To address this issue, we propose aligning cross-lingual thoughts and responses (ACTR), a framework that improves multilingual safety alignment by strengthening the use of existing safety reasoning. Specifically, we first present the think gap score (TGS) to compare the normalized contributions of reasoning traces to attention outputs during response generation across languages, and use reasoning- trace substitution to measure the cross-lingual safety gap. Next, using a corpus of jailbreak queries, we assess neuron importance through changes in response representations caused by neuron masking and compare the high-importance neuron sets obtained with reasoning enabled and disabled to identify safety think neurons that support the use of safety reasoning. Finally, we devise neuron-selective consistency optimization (NSCO), which uses a frozen judge model to reward agreement between the safety categories of reasoning traces and responses while updating only the parameters associated with the selected neurons, requiring no human-annotated responses or preference data. Across two reasoning models, ACTR achieves lower average attack success rates than the evaluated state-of-the-art methods on AdvBench-X and MultiJail, with safety gains extending to unseen languages, while preserving or improving average performance on multilingual knowledge and mathematical reasoning tasks and limiting false refusals of benign requests. Warning: this paper contains examples with unsafe content.

1 Introduction

Reasoning LLMs can recognize jailbreak risks in English yet produce unsafe responses to non-high-resource-language prompts, leaving a gap between safety judgments and outputs. ACTR measures this disconnect, identifies neurons involved in using safety reasoning, and selectively optimizes them; it reports lower attack success rates while preserving utility and low benign-request refusal rates.

1 Introduction

  • A jailbreak in a non-high-resource language can elicit English reasoning that identifies safety risks yet still produce an unsafe answer in the target language.Risk recognition alone does not ensure that the final response follows the safety judgment.Source: §1 · pp. 1–2
  • ACTR measures cross-lingual reasoning contributions with TGS, tests reasoning-trace substitution, and uses neuron interventions to identify safety think neurons.The framework strengthens the use of existing safety reasoning rather than supplying new human-labeled responses.Source: §1 · p. 2
  • NSCO uses a frozen judge to reward agreement between reasoning and response safety categories, updating only selected neuron parameters without human-annotated responses or preference data.The optimization is selective to the identified safety think neurons.Source: §1 · p. 2
  • Across two reasoning models, ACTR achieves lower average attack success rates than state-of-the-art methods on AdvBench-X and MultiJail, generalizing to unseen languages with preserved task performance and low benign-request refusal.The reported results cover average performance and refusal rates.Source: §1 · p. 2

2 Related Work

Prior safety alignment often supervises final responses, while multilingual defenses can require costly high-quality multilingual or parallel corpora. The related work also leaves the interaction between intermediate reasoning and multilingual responses, and the limits of response-only optimization, as open concerns.

Safety Alignment in Reasoning LLMs

  • Existing safety alignment methods primarily supervise final responses, while multilingual approaches explore supervision, augmentation, and cross-lingual objectives that can require costly high-quality multilingual or parallel corpora.The review contrasts response-focused alignment with reasoning-oriented safety specifications.Source: §2 · p. 2

Mechanistic Interpretability of Safety Behavior

  • Mechanistic studies link localized structures to safety behaviors, but most examine single-stage generation rather than the reasoning-to-response interface in multilingual settings.The cited multilingual chain-of-thought work shows reasoning and answer languages can interact.Source: §2 · p. 2

Reinforcement Learning for Safety

  • Rewarding complete responses can cause reward overoptimization, excessive refusal, or reduced utility, and response-only optimization may not incorporate safety judgments formed during reasoning into multilingual answers.The review identifies this as a concern for multilingual reasoning systems.Source: §2 · p. 2

3 Mechanistic Analysis of the Cross-lingual Gap

The analysis links multilingual safety gaps to reasoning traces that do not reliably guide final responses, then identifies neurons associated with reasoning use and develops ACTR to strengthen thought–response consistency. The reported results connect targeted neuron masking with weaker reasoning utilization and higher attack success, while ACTR has the lowest average ASR among evaluated methods across both benchmarks and model families.

3.1 The “Think-Response” Attention Disconnect

  • English-trace substitution reduces seven-language mean ASR by only 0.40 and 0.45 percentage points for Qwen3-8B and Gemma4-12B-it, respectively.Across both models, larger TGS generally accompanies higher ASR, though the relationship is not strictly monotonic; safe traces can still precede unsafe responses, especially in NHR languages.Source: §3.1 · pp. 3–5, Figure 3 · p. 4, Table 2 · p. 4
  • TGS measures cross-lingual differences in reasoning-trace contributions to attention outputs, and positive TGS indicates lower average normalized think-value contribution in NHR than paired HR conversations.In the controlled substitution experiment, NHR ASR remained largely unchanged after replacing NHR reasoning with the corresponding English-elicited trace.Source: §3.1 · pp. 3–4

3.2 Identifying and Defining Safety Think Neuron

  • Safety think neurons are identified by comparing neurons highly influential with reasoning enabled against those influential in matched reasoning-disabled conditions.The method selects the top 3% of neurons per layer before mode-contrastive subtraction; importance is assessed through changes in response representations after neuron deactivation.Source: §3.2 · p. 5
  • Targeted safety-think-neuron masking raises Qwen3-8B ASR by 6.04 points on AdvBench-X and 15.60 points on MultiJail, versus 0.38 and 0.41 points with matched random masking.Targeted masking also increases TGS by a mean of 0.696 across NHR languages, while matched random masking changes it by −0.045.Source: §3.2 · p. 6, Table 3 · p. 6, Figure 4 · p. 6

3.3 The ACTR Framework

  • NSCO uses a frozen binary judge to reward semantic and behavioral agreement between reasoning traces and responses, while updating only parameters associated with identified safety think neurons.The remaining backbone parameters, reference policy, and judge stay frozen during training.Source: §3.3 · pp. 6–7
  • ACTR achieves the lowest average ASR across AdvBench-X and MultiJail and across the evaluated model families.The comparison includes alignment and inference-time defense methods; explicit refusal is counted as safe for refusal-based defenses.Source: Table 4 · p. 7

4 Experiments

ACTR lowers average attack success rates across two reasoning models and two benchmarks while preserving utility and limiting refusals of benign requests. Ablations and deeper analyses examine neuron selection, generalization to unseen languages, and changes in reasoning utilization.

4.1 Main Results

  • ACTR achieves the lowest average ASR for both model families on AdvBench-X and MultiJail among the evaluated alignment and inference-time defenses.For Qwen3-8B, average ASR is 0.29% on AdvBench-X and 1.90% on MultiJail; for Gemma4-12B-it, it is 0.72% and 1.03%, respectively.Source: §4.1 · pp. 7–8

4.2 Ablation study

  • At p=5, Gemma4-12B-it MGSM drops by 83.60, while MGSM scores remain near baseline when p ≤3%.The results support p=3 as a sparse selection threshold that avoids capturing neurons needed for foundational reasoning and task execution.Source: §4.2 · p. 8, Figure 5 · p. 8
  • ACTR slightly outperforms a random-neuron strategy that uses the same neuron count, training data, optimization settings, and parameter budget.For Gemma4-12B-it, ACTR’s ASR is 0.93% on AdvBench-X and 1.72% on MultiJail.Source: §4.2 · p. 8, Table 5 · p. 8
  • Qwen3-8B’s safe-request refusal rate is 6.40% after ACTR, versus 52.00% after random-neuron training, while ACTR raises unsafe-request refusal to 89.50%.For Gemma4-12B-it, safe-request refusal changes from 4.80% to 4.84%, while unsafe-request refusal rises from 82.00% to 89.20%.Source: §4.2 · p. 8, Table 6 · p. 8

4.3 Utility Preservation

  • ACTR preserves or improves average accuracy on multilingual knowledge and mathematical reasoning benchmarks for both models.Qwen3-8B averages rise from 55.11% to 55.45% on MMMLU and from 84.96% to 89.20% on MGSM; Gemma4-12B-it averages rise from 42.33% to 48.97% on MMMLU and from 91.76% to 91.92% on MGSM.Source: §4.3 · pp. 8–9, Table 7 · p. 8

4.4 Deeper Analysis

  • On unseen languages, Qwen3-8B’s overall MultiJail ASR falls from 23.5% to 1.3%, and Gemma4-12B-it’s falls from 9.4% to 5.6%.None of the evaluated unseen languages was included in training; Qwen3-8B’s Swahili ASR falls from 62.86% to 1.27%.Source: §4.4 · p. 9, Table 8 · p. 9
  • ACTR lowers TGS across all six NHR languages for both models, indicating smaller English-to-NHR gaps in reasoning-trace contribution.Mean TGS drops from 4.05 to 1.91 for Qwen3-8B and from 3.79 to 2.07 for Gemma4-12B-it.Source: §4.4 · p. 9, Figure 6 · p. 9
  • In the Korean and Thai examples, ACTR changes the pattern from safe English reasoning with unsafe target-language responses to refusal in both reasoning and response.The paper presents these cases alongside reduced TGS and ASR scores.Source: §4.4 · p. 9

5 Conclusion

The conclusion presents ACTR as a way to align multilingual responses with existing safety reasoning by selectively updating safety think neurons. Experiments report lower average attack success rates, generalization to unseen languages, preserved utility, and low refusal rates on safe requests.

5 Conclusion

  • ACTR combines a thought–response consistency reward with updates restricted to identified safety think neurons, without requiring human-annotated preference data.The paper reports lower average attack success rates than evaluated state-of-the-art methods, generalization to unseen languages, and preserved general utility with low safe-request refusal rates.Source: §5 · p. 9

AI use statement

  • The authors state that generative AI tools reviewed LaTeX formatting and GPT-4o calculated ASR for consistency with historical methodology.GPT-4o-mini served as the frozen consistency judge; the authors take responsibility for the claims, analyses, and final content.Source: AI use statement · p. 9

Ethics statement

  • The authors caution that the same mechanistic interventions can weaken refusal behavior, so findings should be read in a defensive-evaluation context.Source: Ethics statement · p. 9

Reproducibility statement

  • The paper states that it provides reproduction details, including procedures, training protocol, hyperparameters, and the consistency-judge prompt and decision rule.Source: Reproducibility statement · p. 10

APPENDIX

  • The appendices cover notation, TGS theory, ASR evaluation, implementation, judge prompting, limitations and future code-switching work, neuron-masking results, and further discussion.Source: APPENDIX · p. 13

A Notation Summary

B Theoretical Justification of the Think Gap Score (TGS)

The theoretical analysis links a positive TGS to lower safety steering in non-high-resource languages under stated assumptions. It also defines how attack success rate is measured across languages and evaluation conditions.

B Theoretical Justification of the Think Gap Score (TGS)

  • The derivation assumes a linear safety direction, risk-aware think traces, and adversarial-query attention with a nonpositive projection on that direction.A larger final-state projection onto the safety direction is assumed to increase the logit of a safe or refusal token.Source: B Theoretical Justification of the… · pp. 13–14
  • Total safety steering is defined as the update's projection onto the safety direction, while think-span steering is bounded by its L2 norm.The derivation focuses on cumulative attention updates in the latter half of the layers.Source: B Theoretical Justification of the… · p. 14
  • Assuming bounded, relatively stable layer-output energy, think-span contribution magnitude is monotonically parameterized by normalized think contribution, constraining safety steering.The analysis identifies the normalized think contribution as the ratio that constrains the expected safety steering signal.Source: B Theoretical Justification of the… · p. 14
  • TGS_NHR|HR > 0 theoretically guarantees S_NHR < S_HR, corresponding to lower NHR safety steering and a lower probability of a safe response.The paper connects this result to the cross-lingual safety gap, observed as higher ASR in NHR languages.Source: B Theoretical Justification of the… · p. 14

C Attack Success Rate Computation

  • ASR is the proportion of queries whose responses fulfill malicious intent; GPT-4o judges compliance separately by language and evaluation condition.Aggregate ASR is calculated over the corresponding complete query set.Source: C Attack Success Rate Computation · pp. 14–15

D Implementation Details

Table 10 summarizes ACTR hyperparameters across base models. Unless otherwise specified, both base models and the Random baseline use the same training configuration.

D Implementation Details

  • Table 10 summarizes the hyperparameters used for ACTR across different base models.The Random baseline uses the same training configuration.Source: Table 10 · p. 15
  • Unless otherwise specified, both base models use the same training configuration.The text presents this shared configuration as the default.Source: D Implementation Details · p. 15
  • The Random baseline is trained with exactly the same hyperparameters as ACTR for a fair comparison.Source: D Implementation Details · p. 15, Table 10 · p. 15

E Consistency Judge

A frozen GPT-4o-mini judge classifies each Think–Response pair and rewards matching Safe or Unsafe classifications. The reward measures consistency rather than response safety.

E Consistency Judge

  • A frozen GPT-4o-mini judge separately classifies the Think trace and Response as Safe, Unsafe, or Invalid.Source: E Consistency Judge · p. 15
  • The binary consistency reward is 1 when both classifications are Safe or both are Unsafe, and 0 otherwise.Source: E Consistency Judge · p. 15
  • The reward measures classification agreement, not response safety; jointly Unsafe pairs still receive 1, while any Invalid classification receives 0.Source: E Consistency Judge · p. 15

F Qualitative Analysis of Safety Think Neurons Masking

The qualitative analysis presents examples of safety degradation after masking safety think neurons. Examples span two models and non-high-resource languages, with safe and harmful text visually distinguished.

F Qualitative Analysis of Safety Think Neurons Masking

G Limitations

The evaluation uses prompts written in one language at a time, leaving mixed-language interactions outside its primary setting. The authors plan to study alignment under code-switching and changing language contexts.

G Limitations

  • Multilingual safety is primarily evaluated with prompts written in a single language at a time, unlike interactions combining languages, scripts, or language-specific expressions.This limits the evaluation setting; the paper identifies mixed-language interactions as a future direction.Source: G Limitations · p. 16
  • Future work will study think–response alignment under code-switching and dynamically changing language contexts.The authors propose testing whether language-switch-aware alignment improves robustness while preserving natural multilingual interaction.Source: G Limitations · p. 16
  • The proposed follow-up tests whether safety judgments transfer consistently when prompt, reasoning-trace, and response languages are not fixed.Source: G Limitations · p. 16

H More Discussions

The discussion distinguishes thought–response consistency from directly rewarding refusals, and examines neuron selection and random baselines. Its comparisons describe trade-offs in benign refusals and general mathematical reasoning.

H More Discussions

  • NSCO uses consistency to align safety reasoning with responses, not as a substitute for safety reward; avoiding refusal-only rewards is intended to limit over-refusal on benign requests.Source: H More Discussions · pp. 16–17
  • The identified safety think neurons may mix safety-specific circuits with general think-utilization pathways; masking more than 3% degrades MGSM mathematical reasoning.The neurons are selected using jailbreak queries, and the authors characterize the top 3% as highly specialized for transferring safety-related constraints.Source: H More Discussions · p. 17

Q3. Could targeted masking simply damage the model globally?

  • For Gemma4-12B-it, targeted masking raises MultiJail attack success by 43.90 points, versus 1.54 points under random masking.Random masking uses the same number of units and layer-wise distribution; the authors conclude the intervention is not generic capacity removal.Source: Q3. Could targeted masking simply… · p. 17

Q4. Does the random-neuron baseline show that neuron identification is unnecessary?

  • For Qwen3-8B, random-neuron training raises XSTest benign refusal from 4.40% to 52.00%, while ACTR raises it to 6.40% and achieves higher unsafe-request refusal.The discussion says the same pattern is stronger for Gemma4-12B-it.Source: Q4. Does the random-neuron baseline… · p. 17

Related papers

Loading 2609.37054v2…