PathSeeker: Exploring LLM Security Vulnerabilities with a Reinforcement Learning-Based Jailbreak Approach

1 Introduction

• Existing black-box methods may rely on proxy-generated harmful answers, limiting creativity or usefulness when proxy and target models have different unsafe behaviors.GPTFuzzer also does not leverage experience gained across attack attempts.Source: §1 · p. 2
• Its reward mechanism uses response vocabulary richness and answer maliciousness as feedback, without requiring harmful reference answers.The authors observed vocabulary expanding as the target model shifted from refusing harmful questions toward answering them.Source: §1 · p. 3
• The authors cross-validated GPT-4o-mini-based attack judgments with Llama Guard3 and report high consistency between the evaluations.They also analyzed the mutators, reward mechanism, and model selection, and tested transfer to Llama 3 models, including a 405-billion-parameter model.Source: §1 · p. 3
• PathSeeker uses smaller reinforcement-learning agents to learn from target-model attack trials and guide changes to harmful questions and jailbreak templates.The approach treats attack attempts as steps through a target-specific security maze, guided by feedback and accumulated experience.Source: §1 · pp. 2–3
• PathSeeker achieved the highest average attack success rate among five state-of-the-art baselines across 13 open- and closed-source models, especially strongly aligned commercial LLMs.The authors also report transfer to Llama 3 models, including a 405-billion-parameter model.Source: §1 · pp. 3–4

2.1 Prelimary

• Reinforcement learning repeatedly updates an agent's strategy through actions, environmental feedback, and cumulative rewards.The paper presents this trial-and-error process as suitable for complex, dynamic environments.Source: §2.1 · p. 4

2.2 Overview

• PathSeeker uses multiple agents to choose question and template mutations from input-output state, then combines the mutated inputs for the target LLM.A judgment model assesses the response; its confidence score and Information Quantization (IQ) provide rewards, and successful templates return to the template pool.Source: §2.2 · p. 4

2.3 Challenges

• The action space covers changing both the harmful question and jailbreak template while preserving harmful attributes and diversifying input styles.The authors intend this diversity and ambiguity to test model stability under varied inputs.Source: §2.3 · p. 5
• A judgment model automatically assesses attack success, avoiding repeated human evaluation during reinforcement-learning training.The authors describe human judgment as time-consuming and impractical across multiple training sessions.Source: §2.3 · p. 5
• For closed-source models, PathSeeker represents system state with the input and final generated text because internal weights, hidden states, and logits are inaccessible.The authors also note that accessing those internals in open-source models requires extra hardware resources and time.Source: §2.3 · p. 5
• Binary success or failure rewards miss progress before an attack succeeds, so IQ measures vocabulary richness in response clauses related to the question.The authors propose using IQ to reward gradual movement toward success during the attack process.Source: §2.3 · p. 6

• Jailbreak templates are selected using UCB scores based on their past performance in successful attacks.Harmful questions, by contrast, are selected randomly.Source: §2.4 · p. 6
• With probability 0.95, PathSeeker selects the highest-scoring template; with probability 1 − 0.95, it selects randomly.The authors say this random selection supports exploration of the global optimum.Source: §2.4 · p. 6
• The template-selection strategy prioritizes templates more likely to succeed while reducing use of ineffective ones.Random selection remains part of the policy to provide exploration.Source: §2.4 · p. 6

2.5 Template Mutator and Question Mutator

• The template mutator uses Generate, Crossover, Expand, Shorten, and Rephrase operations to create or revise jailbreak templates.The operations respectively create a similar-style scene, merge template seeds, add detail, compress, or restructure wording without changing semantics.Source: §2.5 · p. 6
• The question mutator changes tone, adds confusion, splits or restructures the question, and substitutes synonyms while aiming to preserve harmful semantics.The authors also describe making mutated questions appear deceptive from a human perspective.Source: §2.5 · pp. 6–7
• PathSeeker selects GPT-4o-mini as its mutation model after considering time efficiency, performance, and cost.The authors describe the selection as cost-effective while ensuring efficiency and high quality.Source: §2.5 · p. 7

• The judgment model classifies responses as full refusal, full compliance, partial refusal, or partial compliance, each with a confidence score.The authors use its feedback to accelerate reinforcement learning and automate the attack process.Source: §2.6 · p. 7
• The judgment model is intended to provide rapid feedback and eliminate human intervention in the attack process.Its prompt guides an LLM to assess attack outcomes.Source: §2.6 · p. 7
• The authors select GPT-4o-mini as the judgment model after comparing several commercial and open-source LLMs for accuracy and API cost.They report that commercial models excelled at identifying and filtering potentially harmful input content.Source: §2.6 · p. 7

2.7 Information Quantification

• Guided prompts can move a target from refusing a harmful question toward offering relevant suggestions.The section describes this progression as a motivation for measuring information in responses.Source: §2.7 · pp. 7–8
• The method extracts response subsentences related to the input, then counts their nouns, verbs, adjectives, and adverbs.GPT4-mini performs extraction; a Counter tallies the four parts of speech in each subsentence.Source: §2.7 · p. 8
• Richer responses may contain answers to harmful questions as the model gradually relaxes its safety constraints.The authors use increasing response content as a way to assess information gained during unsuccessful attack attempts.Source: §2.7 · p. 8

2.8 Multiple-Agent Reinforcement Learning

• Question and template agents select mutations whose combined outputs are sent to the target; judged harmfulness and response information contribute to reward and state.The input prompt and target response, plus embedded harmful inputs, inform the agents' decisions.Source: §2.8 · pp. 8–9

2.8.1 Action Agents.

• Each Actor-Critic agent chooses among five mutation operations, and MADDPG jointly optimizes their decisions while accounting for the other agent’s choices.Question operations are Euphemize, Confusion, Split, Restructure, and Substitution; template operations are Generate, Crossover, Expand, Shorten, and Rephrase.Source: §2.8.1 · p. 9

2.8.2 Embedding LLM State and Malicious Inputs.

• The agents receive embeddings of the prompt-response state and harmful question-template inputs because they cannot directly process text or access model internals.PathSeeker uses all-MiniLM-L6-v2 to embed both text pairs for the agents’ MLPs.Source: §2.8.2 · p. 9

2.8.3 Reward.

• The reward combines response-information and judged-harmfulness rewards, with alpha set to 0.5 to balance them.Each component rewards an increase and penalizes a decrease from the previous value, using a logarithmic scale.Source: §2.8.3 · p. 10

3.1 Research Questions

• Five research questions cover mutation-model choice, reinforcement learning, reward and action design, comparisons with black-box attacks, and transfer attacks.The evaluation also compares multi-agent and single-agent reinforcement-learning designs.Source: §3.1 · pp. 10–11

3.2.1 Target models.

• Targets span closed-source and open-source models, while the RQ4 comparison uses 13 models and RQ5 tests transfer across the Llama-3.1 series.The RQ1–RQ3 studies use selected targets, including Llama-2-7b-chat, GPT-3.5-turbo, and GLM-4-air.Source: §3.2.1 · p. 11

3.2.2 Mutation Model.

• Five mutation-model candidates are compared for attack performance, cost, and processing time to select a model for iterative mutation.The candidates are GPT-4o-mini, Deepseek-chat, Llama-3.1-70b, Qwen2-72b-instruct, and Vicuna-7b.Source: §3.2.2 · p. 11

3.2.3 Datasets.

• The experiments use 100 harmful questions and 77 initial jailbreak templates drawn from the GPTFuzzer datasets and selections.The questions were author-crafted or crowdsourced; the templates were selected by GPTFuzzer.Source: §3.2.3 · p. 11

3.2.4 Baselines.

• Five open-source baselines include ReneLLM (Ding et al., 2024), Jailbroken (Yuan et al., 2024), GPTFuzzer (Yu et al., 2023), CodeChameleon (Lv et al., 2024), and Cipher (Yuan et al., 2023).The authors omit RL-JACK and RLTA because they were not open source at the time.Source: §3.2.4 · pp. 11–12

3.2.5 Evaluation Metrics.

• Top1-ASR uses the best template on 100 questions, while Top5-ASR counts success if any of the five best templates works; GPT-4o-mini judges outcomes.Success requires a non-refusal and harmful or otherwise inappropriate content; Llama Guard cross-validation and manual inspection of 50 results are also reported.Source: §3.2.5 · p. 12

4.1 Choice of Mutation Model for Mutators (RQ1)

• GPT-4o-mini was selected as the mutation model for its balance of attack performance, iteration time, and cost.Qwen2-72b-instruct had the highest Top1 and Top5 ASR, while GPT-4o-mini’s mutation time was substantially shorter than Deepseek-chat’s and Qwen2-72b-instruct’s.Source: §4.1 · p. 13, Table 1 · p. 13

4.2 Impact of Reinforcement Learning (RQ2)

• Multi-agent MADDPG outperforms both non-reinforcement learning and single-agent DQN configurations on attack success rates across the two tested targets.The comparisons used GPT-3.5-turbo and Llama-2-7b-chat over 3,000 rounds.Source: §4.2 · pp. 13–14, Table 2 · p. 14

4.3 Design of Reinforcement Learning (RQ3)

• Using only either reward component lowers ASR, and mutating both questions and templates improves ASR over mutating templates alone.The comparisons used Llama-2-7b-chat and GLM-4-air; the authors report that each component contributes to full PathSeeker’s effectiveness.Source: §4.3 · p. 14, Table 3 · p. 14

4.4 Comparison with Other Attack Methods (RQ4)

• PathSeeker achieves the best Top1-ASR in 13/13 comparisons and best Top5-ASR in 11/13; its reported average Top1-ASR is 98.9% and Top5-ASR is 99.7%.GPTFuzzer has the best Top5-ASR on two models; the authors report PathSeeker as best or close to best under Llama Guard evaluation as well.Source: Table 4 · p. 15, §4.4 · pp. 15–16

4.5 Transfer Attack (RQ5)

• Template transfer success varied by target, from 97% Top1 and 98% Top5 on Llama-3.1-8b to 64% Top1 and 81% Top5 on Llama-3.1-405b; agent transfer performed better than non-transfer.The authors note potential for agent transfer, especially from smaller to larger models within the same series.Source: §4.5 · pp. 16–17, Table 6 · p. 16

5 Related Works

• Jailbreak attacks seek to bypass LLM security and ethical guidelines, with white-box methods accessing model internals and black-box methods relying on observed inputs and outputs.The section organizes prior approaches around this access distinction.Source: §5 · p. 17
• White-box methods exploit target-model internals: GCG combines greedy and gradient-based search, while AmpleGCG is limited against robust defenses such as GPT-4.AutoDAN is also described as a gradient optimization-based attack.Source: §5 · p. 17
• Black-box approaches include prompt rewriting and contextual nesting in ReNeLLM, encrypted queries in CodeChameleon, and attacker-target collaboration in PAIR.Other examples include Jailbroken, CipherChat, defense reverse engineering, and GPTFUZZER's seed-template mutations.Source: §5 · p. 17

6 Discussion

• Evaluation uses GPT-4o-mini harm judgments, cross-validated with Llama Guard3, across 13 models and 100 harmful questions from the GPTFuzzer dataset.The authors identify judgment errors as an internal validity threat and model/question selection as an external validity concern.Source: §6 · p. 18
• PathSeeker uses fewer overall queries than GPTFuzzer but takes longer because each iteration includes reinforcement-learning strategy selection and template and question mutation.The authors also say effectiveness depends heavily on manually crafted initial jailbreak templates.Source: §6 · p. 18
• Randomness in LLM-based components and reinforcement-learning agents can cause variations in attack effectiveness.The authors trace randomness to mutators, judgment, information quantization, agent initial states, optimization, and outputs from other components.Source: §6 · p. 18
• Attack strategies and effectiveness may not fully transfer across model types, particularly when models defend against PathSeeker's mutation strategies.Source: §6 · p. 18
• The authors propose using PathSeeker to test model security and enhance jailbreak and harmful-question datasets; defenses may target its action-space features or response-vocabulary richness.They suggest limiting response vocabulary as one defense direction.Source: §6 · p. 18

7 Conclusion

• PathSeeker uses multi-agent reinforcement learning to iteratively refine attack strategies and identify mutation actions for navigating LLM security defenses.Source: §7 · p. 18
• Across 13 LLMs, PathSeeker outperforms five existing SOTA jailbreak methods in attack success rate and efficiency, especially on commercial models with strong security alignment.Source: §7 · p. 19
• A vocabulary-richness reward encourages more response content, while a double-pool mutation strategy aims to improve attack diversity and effectiveness.The paper says these components work together to increase attack success rate.Source: §7 · p. 19