PaperlayerView exact revision on arXiv

Source-linked AI summary

TACS: Trajectory-Aware Candidate Selection for LLM Jailbreak Suffix Optimization

Shiliang Xiao

arXiv:2608.29564v1cs.CLcs.LG

TL;DR

Myopic current-loss selection can reward candidates that look better immediately but yield worse downstream jailbreak outcomes, making candidate selection a bottleneck. TACS uses trajectory-aware scoring with reference-policy regularization and discriminator-guided correction, and it consistently outperforms strong baselines under the same search budget with higher ASR and more stable trajectories.

  • Problem

    Current-loss candidate selection can overfit short-horizon proxy gains, producing selection-stage reward hacking and a mismatch with downstream jailbreak outcomes.

  • Method

    TACS replaces naive retention with trajectory-aware scoring, reference-regularized correction, and discriminator-guided adjustment using observable candidate signals.

  • Results

    TACS consistently outperforms strong baselines under the same search budget, achieving higher attack success rates and more stable optimization trajectories.

  • Takeaways & Limitations

    Mitigating selection-stage reward hacking caused by myopic candidate selection is critical for improving jailbreak suffix optimization.

  • Takeaways & Limitations

    TACS does not query additional intermediate model outputs and does not assume access to a true reward beyond the standard optimization proxy.

Abstract

from arXiv · show

Gradient-based jailbreak suffix optimization methods typically update the suffix by retaining the candidate with the lowest current loss. We show that this seemingly natural design is fundamentally myopic: candidates that look better under the current-step proxy often fail to produce better jailbreak outcomes later in the search, revealing a form of selection-stage reward hacking. This suggests that candidate selection, rather than candidate generation alone, is a hidden bottleneck in suffix optimization. To address this issue, we propose \OURS{}, a trajectory-aware candidate selection framework for jailbreak suffix optimization. Instead of selecting candidates solely by their immediate loss, \OURS{} augments per-step evaluation with a trajectory-aware proxy and stabilizes selection with reference-policy regularization and a discriminator-estimated chi-squared correction, encouraging choices that remain effective beyond the current step. Experiments on HarmBench show that \OURS{} consistently outperforms strong baselines under the same search budget, substantially improving attack success rates while exhibiting more stable optimization behavior throughout the search. Our findings highlight that mitigating selection-stage reward hacking caused by myopic candidate selection is critical for improving jailbreak suffix optimization.

1 Introduction

Gradient-based jailbreak suffix optimization exposes a selection-stage reward-hacking problem: current-step loss can improve while downstream attack success remains poor. TACS addresses this myopia with trajectory-aware, reference-regularized candidate selection and improves performance and stability under matched search budgets.

  • Safety-aligned LLMs remain vulnerable to jailbreak attacks, which provide a practical testbed for auditing alignment mechanisms.
  • Prior jailbreak research has emphasized attack strength more than whether the signal steering optimization reliably reflects the true objective.
  • Training loss can decrease steadily while ASR stays low and rises in delayed, irregular bursts, exposing a loss–ASR mismatch.
  • Standard selection retains the lowest-current-loss candidate, overemphasizing immediate proxy gains and steering the search toward locally attractive but globally suboptimal updates.
  • TACS combines trajectory-aware and reference-regularized signals to favor candidates that remain useful for subsequent optimization.
  • Nearly 10% average ASR improvement on HarmBench accompanies stronger performance and more stable optimization trajectories under the same search budget.

2 Methodology

TACS replaces myopic lowest-current-loss retention with trajectory-aware scoring, reference regularization, and discriminator-guided correction. It evaluates candidates within each gradient-proposed batch and retains the candidate with the lowest corrected energy.

  • Optimization Setup: TACS optimizes a universal jailbreak suffix over T steps, generating a candidate batch from the current suffix with a standard gradient-based proposer.At each step, candidates are evaluated before selecting the next suffix.
  • Optimization Setup: Current attack loss measures each candidate’s average target-token cross-entropy across the training behaviors.TACS also retains an auxiliary early-token loss that captures whether the candidate supports the initial jailbreak scaffold.
  • Trajectory-Aware Candidate Scoring: TACS adds a one-step improvement proxy because selecting only the lowest current loss can favor edits that fit the present step but leave little room for further progress.The proxy is estimated from candidate-specific gradient statistics tied to edited positions and replacement tokens.
  • Reference-Regularized Candidate Selection: The trajectory-aware scores are converted into a batch-induced selection distribution, while a reference distribution provides stabilizing preferences over the same candidates.The reference rule omits the one-step bonus and is intended to prevent collapse onto sharp, batch-specific proxy preferences.
  • Discriminator-Guided Correction: A discriminator compares candidates sampled from the current and reference distributions, producing a correction that penalizes candidates overly favored by current selection bias.Its features include current loss, loss dispersion, early-token loss, and gradient-derived statistics.
  • Final Selection Rule and Practical Optimization Loop: The practical loop computes candidate losses and the look-ahead proxy, constructs both distributions, applies correction, and retains the candidate with the lowest corrected energy.During warm-up, TACS uses greedy current-loss selection before the discriminator has accumulated enough labeled samples.

3 Experiment

The experiments evaluate TACS against GCG and GJO on HarmBench under matched search settings, measuring source- and target-model ASR, component contributions, proxy mismatch, and transfer behavior. TACS achieves stronger source and transfer performance while producing more sustainable optimization trajectories.

  • Experimental Setup: The evaluation uses HarmBench with a 500-step budget, two target tokens, candidate batch size 128, and top-k proposal size 256.ASR is reported on the source model and target models; source-side ASR is S-ASR and target-side average ASR is T-ASR.
  • Overall Results: TACS reaches 96.0 S-ASR on Llama3, exceeding GJO (61.5) and GCG (20.5) under the same search budget.The reported comparison is for searches on the Llama3 source model.
  • Overall Results: TACS also achieves the best average T-ASR, including 85.5 versus 69.0 on Qwen2-7B, 68.0 versus 40.0 on Yi-1.5-9B, and 14.5 versus 4.0 on Gemma-7b.The passage describes these as transfer gains that do not come at the cost of source-side performance.
  • Ablation Study: Removing either the trajectory-aware term or reference-guided correction reduces source- and target-side performance, with the larger drop consistently occurring without the trajectory-aware term.On Llama3, target-side average ASR falls from 50.3 to 42.7 without trajectory awareness and to 47.5 without reference correction.
  • Proxy Mismatch: After roughly step 350, GCG and GJO continue reducing loss while ASR largely plateaus, whereas TACS raises ASR earlier and continues improving later.TACS’s loss is not always lowest, but its trajectory-aware selector retains candidates useful for subsequent optimization.
  • Case Study: Table 3 shows GJO drifting into refusals, vague prefatory text, or title-style responses, while TACS preserves a more consistent step-by-step scaffold across source and target models.The case study compares transfer behavior for the same harmful query.

Model GJO TACS

The transfer case study contrasts GJO and TACS across source and target models. TACS tends to preserve candidate behavior whose early-token pattern and short-horizon improvement remain aligned over time.

  • Model GJO TACS: The case study compares GJO on the left with TACS on the right across source and target models using truncated response snippets.The snippets are shown for safety.
  • Model GJO TACS: TACS tends to retain candidates whose early-token behavior and short-horizon improvement signal remain aligned over time.

4 Related Work

Related work spans black-box and white-box jailbreak attacks, with gradient-based suffix optimization forming a representative white-box direction. Prior methods improve search diversity, efficiency, attention manipulation, or transferability.

  • White-box and Black-box Jailbreak Attacks: Black-box jailbreak methods rely on model queries and responses, including iterative attacker-LLM refinement, branch pruning, and automated prompt generation.
  • Gradient-Based Suffix Optimization Attacks: GCG establishes a standard discrete gradient-based suffix optimization pipeline for white-box jailbreak attacks.
  • Gradient-Based Suffix Optimization Attacks: I-GCG adds diverse target templates, adaptive multi-coordinate updates, and easy-to-hard initialization, while another method uses adaptive dense-to-sparse constrained search.
  • Gradient-Based Suffix Optimization Attacks: AttnGCG manipulates attention patterns related to safety prompts, and GJO revisits the optimization objective to improve suffix transferability.

5 Conclusion

The paper identifies selection-stage reward hacking as a core issue in gradient-based jailbreak suffix optimization and proposes TACS to address it. HarmBench experiments report higher attack success rates and more stable trajectories under the same search budget.

  • 5 Conclusion: Standard loss-based retention can select candidates that look best under the current proxy but lead to worse downstream jailbreak outcomes, creating a loss–ASR mismatch.
  • 5 Conclusion: TACS combines trajectory-aware scoring with reference-regularized correction to improve candidate retention decisions.
  • 5 Conclusion: Under the same search budget, TACS consistently outperforms strong baselines with higher attack success rates and more stable optimization trajectories.
Loading 2608.29564v1…