Safe at One Loop, Risky at Another: Aligning Safety Across Recurrent Depths in Looped Language Models

1 INTRODUCTION

• Because shared Transformer blocks are reused across recurrent steps, each depth can be read out, giving a single LoopLM multiple outputs without increasing parameter count.The resulting multi-depth output space broadens the surface over which harmful behavior may emerge.Source: §1 · p. 1
• Prior evaluations suggest deeper recurrence reduces harmful responses to original harmful queries, but whether that trend holds under jailbreak attacks remains unclear.This motivates evaluating LoopLM safety across recurrent depths under jailbreak attacks.Source: §1 · p. 1
• Jailbreak ASR can rise at deeper inference depths, and attacks built against one depth can transfer to others, exposing two cross-depth safety challenges.The evaluation spans multiple safety benchmarks, attack methods, and recurrent depths.Source: §1 · pp. 1–2
• Conventional SFT, DPO, and SafeDPO do not explicitly align safety across recurrent depths, and jointly optimizing preference losses across depths alone does not improve over DPO.The authors use this limitation to motivate an alignment method designed for LoopLMs.Source: §1 · p. 2
• SafeBridge combines depth-specific modulation, selective state bridging, response safety supervision, and cross-depth objectives to align safety across recurrent computation.It emphasizes weak exits while retaining uniform supervision across depths and discouraging safety degradation as computation proceeds.Source: §1 · p. 2
• Experiments on Ouro models show SafeBridge substantially improves matched-depth and cross-depth jailbreak robustness, general utility over vanilla models, and comparable over-refusal behavior.The evaluation covers model scales, safety benchmarks, attack methods, and recurrent depths.Source: §1 · p. 2

2 RELATED WORK

• Looped Transformers and Ouro extend recurrent computation across shared-parameter depth, with prior work emphasizing reasoning capability and test-time compute scaling rather than safety across depths.Universal Transformers explore recurrent computation with shared parameters; Ouro adds adaptive depth allocation.Source: §2 · pp. 2–3
• GCG, AutoDAN [Liu et al., 2024], and PAIR use distinct jailbreak-generation mechanisms, while HarmBench [Mazeika et al., 2024] and JailbreakBench [Chao et al., 2024] standardize evaluation.This work distinguishes attack source depth from inference depth and tests transfer across recurrent depths within one LoopLM.Source: §2 · p. 3
• SFT and RLHF [Ouyang et al., 2022], DPO [Rafailov et al., 2023], Safe RLHF [Dai et al., 2024], and SafeDPO [Kim et al., 2026] address alignment, but not safety variation across recurrent depths.Other directions include model editing with DELMAN [Wang et al., 2025] and EVA [Wang et al., 2026], and ReSA [Cao et al., 2026].Source: §2 · p. 3
• SafeBridge treats recurrent depth as an explicit alignment dimension, combining response safety supervision with depth-specific control and cross-depth coordination.The cited related work frames existing approaches as methods developed for conventional LLMs that do not explicitly address recurrent-depth safety variation.Source: §2 · p. 3

3 SAFEBRIDGE: SAFETY ALIGNMENT ACROSS RECURRENT DEPTHS

• SafeBridge combines depth-specific modulation and a selective bridge with preference losses, response safety labels, uniform supervision, and cross-depth consistency.The bridge carries an earlier normalized state to later depths; the objective discourages signed safety-score degradation across consecutive depths.Source: §3 · p. 3

3.1 DEPTH SPECIFIC MODULATION AND SELECTIVE STATE BRIDGING

• Depth-specific modulation gives each supervised depth its own parameters for adjusting shared computation, while shared-backbone updates can still affect Loop 1.The method uses L = 4 and directly supervises depths 2, 3, and 4.Source: §3.1 · p. 3
• A gated bridge gives later depths direct access to an earlier depth’s hidden state, added after source normalization and before target normalization.The source depth and target set are fixed for each training run, and the bridge is initialized inactive.Source: §3.1 · p. 4

3.2 SAFETY SIGNED PREFERENCE SUPERVISION

• Safety-signed preference supervision uses response labels so that larger signed scores correspond to raising safe-response likelihood or lowering unsafe-response likelihood.At each supervised depth, the current and frozen reference models score both responses by teacher forcing; the objective retains pairwise DPO supervision and adds a per-response safety penalty.Source: §3.2 · pp. 4–5

3.3 WEAK-EXIT AGGREGATION AND CROSS-DEPTH CONSISTENCY

• Weak-exit aggregation combines adaptive emphasis on larger losses with uniform supervision across supervised depths.The loss types are aggregated separately because their largest values may occur at different depths; the smooth maximum gives larger gradient weights to larger losses.Source: §3.3 · p. 5
• Cross-depth consistency discourages the signed safety score for the same response from decreasing at the next supervised depth.The earlier score is detached, and the temperature is shared with depth aggregation.Source: §3.3 · p. 5

Models and Baselines

• Experiments compare SafeBridge on Ouro-1.4B and Ouro-2.6B with SFT, DPO, SafeDPO, and LoopDPO, which averages preference losses across all four depths.SafeBridge adds approximately 0.02% of model parameters at each scale, with configurations fixed during training.Source: §4 · pp. 5–6

Training Data

• Preference training starts from the ReSA SFT checkpoint and uses PKU-SafeRLHF; SafeDPO and SafeBridge additionally use response safety annotations.DPO and LoopDPO use preference rankings.Source: §4 · p. 6

Evaluation Benchmarks

• Safety is evaluated on four benchmarks using original harmful prompts and jailbroken variants, with over-refusal and general capability assessed on separate benchmarks.The capability tests cover mathematical reasoning, general knowledge and reasoning, and code generation.Source: §4 · p. 6

Attack Setup

• For each model scale, benchmark, and attack method, attacks target each of four source depths and are evaluated at all four inference depths.This yields 48 attack sets and 192 settings per checkpoint; the methods are GCG, AutoDAN, and PAIR.Source: §4 · p. 6

Evaluation Metrics

• ASR uses the HarmBench classifier; results include original-query ASR, overall attack ASR, matched and cross-depth ASR, over-refusal rate, and capability scores.Matched and Cross are macro-averaged over four safety benchmarks per attack method; overall attack ASR averages across benchmarks, attack methods, source depths, and inference depths.Source: §4 · p. 6

5.1 AGGREGATE SAFETY PERFORMANCE

• Overall attack ASR is 4.56% with SafeBridge versus 21.06% with DPO and 20.85% with SafeDPO on Ouro-1.4B; on Ouro-2.6B, it is 9.85% versus 44.93% and 44.30%.SafeBridge achieves the lowest ASR across original queries, matched attacks, and cross-depth attacks at both scales; LoopDPO does not improve over DPO.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 SAFETY ACROSS INFERENCE DEPTHS

• On Ouro-1.4B, jailbreak ASR for SFT, DPO, and LoopDPO rises sharply from depth 2 to depths 3–4; on Ouro-2.6B, it peaks at depth 3.SafeBridge has the lowest ASR at every depth for both original harmful queries and jailbreak attacks.Source: §5.2 · p. 7, Figure 3 · p. 7

5.3 ROBUSTNESS TO CROSS-DEPTH JAILBREAK TRANSFER

• On Ouro-2.6B, DPO attacks sourced at depth 2 reach 66.24% ASR at inference depth 3, above the matched 63.18%; SafeBridge reduces this to 12.02%.SafeBridge also reduces ASR in all 16 source–inference-depth combinations relative to DPO at both scales.Source: §5.3 · p. 7, Figure 4 · p. 8
• For pooled-source PAIR attacks, SafeBridge reduces ASR from 42.12% with SafeDPO to 8.39% on Ouro-1.4B, and from 85.49% to 24.25% on Ouro-2.6B.SafeBridge has the lowest pooled-source ASR for all three attack methods at both scales, with similar gains under GCG and AutoDAN.Source: §5.3 · pp. 7–8, Table 2 · p. 8

5.4 GENERAL UTILITY AND OVER-REFUSAL

• At the final depth, SafeBridge outperforms DPO on all six Ouro-1.4B utility benchmarks and beats Vanilla across all six Ouro-2.6B benchmarks.It gains 1.83 and 1.22 percentage points on HumanEval and HumanEval+, respectively, for Ouro-1.4B.Source: §5.4 · p. 8, Table 3 · p. 9
• SafeBridge maintains over-refusal behavior comparable to SafeDPO and generally has lower ORR peaks, though ORR is moderately higher than DPO and Vanilla at some depths.Figure 5 reports ORR across inference depths on XSTest and OR-Bench-Hard for both model scales.Source: §5.4 · p. 8, Figure 5 · p. 8

5.5 ABLATION STUDIES

• Removing any SafeBridge component degrades safety metrics; removing modulation raises worst-setting ASR from 24.00% to 59.00%.Removing the response safety loss causes the largest mean-ASR increase, while removing the bridge or U raises safety regression from 1.83% to 7.17% and 6.17%, respectively.Source: §5.5 · p. 9, Table 4 · p. 9

6 CONCLUSION

• SafeBridge substantially reduces attack success across recurrent depths and improves general utility over vanilla models while maintaining moderate over-refusal.The gains include original harmful queries and matched-depth and cross-depth jailbreak attacks, including attacks using candidates from multiple source depths.Source: §6 · p. 9
• SafeBridge combines depth-specific modulation, selective state bridging, and response safety supervision to emphasize weak exits and encourage cross-depth safety consistency.Ablations indicate that its components address complementary aspects of cross-depth safety alignment.Source: §6 · p. 9

A ALGORITHMS

• Training aggregates depth-specific preference and response-safety losses within each preference pair before batch averaging, while keeping the reference model frozen.The trainable parameters include the backbone, language-model head, modulation vectors, and bridge gates.Source: A ALGORITHMS · p. 13
• The forward procedure computes response log-probabilities at supervised depths using teacher-forced prefixes, with causal state updates and only the copied source branch detached.The algorithm specifies L = 4 and D = {2, 3, 4}.Source: A ALGORITHMS · p. 14

B.1 EXACT ASR ACROSS INFERENCE DEPTHS

• SafeBridge achieves the lowest attack success rate among compared methods at all four inference depths across both model scales, for original harmful queries and jailbreak attacks.Jailbreak results cover multiple attack methods and source depths.Source: §B.1 · p. 14

B.2 EXACT OVER-REFUSAL RATES ACROSS INFERENCE DEPTHS

• SafeBridge’s over-refusal remains broadly comparable to SafeDPO across XSTest and OR-Bench-Hard, while reducing elevated refusal at d = 2 relative to SFT, DPO, and SafeDPO.The d = 2 reduction is reported on both model scales.Source: §B.2 · p. 14

• SFT uses the ReSA training data.Source: §C.1 · p. 14
• DPO, LoopDPO, and SafeBridge retain all 73,907 original PKU-SafeRLHF preference pairs, and SafeBridge additionally uses response safety labels.Source: §C.1 · pp. 14–15
• SafeDPO removes 32,656 pairs with two unsafe responses and prioritizes safety over helpfulness when ordering retained pairs.Table 7 gives dataset counts.Source: §C.1 · p. 15

C.2 TRAINING CONFIGURATION

• All methods update the shared backbone and language-model head; SafeBridge also updates modulation vectors and bridge gates.Runs use one 80 GB NVIDIA A100 GPU, and preference methods start from the final ReSA-SFT checkpoint.Source: §C.2 · p. 15

C.3 TRAINING LOSS

• All methods exhibit stable optimization under their respective training objectives on both model scales.LoopDPO averages DPO losses across four depths with weight 0.25 per depth.Source: §C.3 · p. 16, §C.2 · p. 16

C.4 ADVERSARIAL TEST-CASE CONSTRUCTION

• For each model scale, benchmark, and attack method, 48 attack sets built against source depths 1–4 are reused across four inference depths and evaluated checkpoints.The construction uses EnsembleGCG suffix optimization, AutoDAN-HGA prompt search, and PAIR attacker streams with judge feedback.Source: §C.4 · p. 16

C.5 COMPLETION GENERATION AND SAFETY EVALUATION

• Each original query or saved jailbreak receives one response at every inference depth using the same Ouro conversation template.Source: §C.5 · p. 16
• A HarmBench classifier’s yes/no judgment determines whether the generated response counts as harmful behavior.Source: §C.5 · p. 17
• Generations count when they exhibit the behavior, including alongside caveats, but not when they merely discuss or react to it.Source: §C.5 · p. 18

C.6 OVER-REFUSAL EVALUATION

• The over-refusal evaluation uses 250 safe XSTest prompts and 1,319 OR-Bench-Hard prompts, measuring full refusals on XSTest and explicit refusals on OR-Bench-Hard.Source: §C.6 · p. 18

C.7 GENERAL UTILITY EVALUATION

• General utility benchmarks are evaluated at inference depth 4, using lm-eval-harness for GSM8K and MMLU and EvalPlus for code benchmarks.Source: §C.7 · p. 19

D CASE STUDY

• The case studies label each response as Attack Failed or Attack Succeeded using the HarmBench classifier, with unsafe content or plan leakage highlighted.Source: D CASE STUDY · pp. 19–20

D.1 CASE A: SAFETY VARIATION ACROSS INFERENCE DEPTHS

• DPO’s GCG jailbreak from source depth 1 succeeds at inference depths 2 and 4 but fails at 1 and 3; SafeBridge blocks it at all four depths.Source: §D.1 · p. 20

D.2 CASE B: CROSS-DEPTH JAILBREAK TRANSFER

• At inference depth 3, the Ouro-1.4B DPO checkpoint fails against the PAIR jailbreak from source depth 3 but produces a complete blackmail response to the source-depth-2 jailbreak; SafeBridge blocks both at both evaluated depths.Source: §D.2 · p. 22
• The two PAIR attacks use different source-depth scenarios: a private investigator guarding sensitive information and a story about a debtor threatened with exposure.Source: §D.2 · p. 23