Source-linked AI summary
AlcaTRAz - Anchored Tree-Rule Defense Against Jailbreaks
Jakub Reš, Petr Kaška, Martin Perešíni, Martin Ukrop, Kamil Malinka
TL;DR
Jailbreak prompts can bypass model safeguards, while many defenses depend on model internals that are unavailable in black-box deployments. AlcaTRAz learns a prompt-only rule-tree transformation; across evaluated models and attacks, it improves the joint security-functionality outcome while leaving residual risk.
Problem
Jailbreak prompts can bypass safeguards, and many defenses require access to model internals, limiting their applicability to black-box deployments.
Source: §1 · p. 2Method
Results
73.4 % of model-attack combinations favor AlcaTRAz on composite score S; the modal judge score shifts from 10 undefended to 2 after defense.
Source: §8 · p. 16Takeaways & Limitations
AlcaTRAz is positioned as one mitigation layer in a defense-in-depth strategy, not a standalone guarantee.
Source: §8 · p. 16The evaluation uses fixed attacks and does not test defense-aware or rule-aware adaptive attackers.
Source: §7 · p. 15
Abstract
from arXiv · showhide
Large language models (LLMs) are vulnerable to jailbreak attacks that bypass safety alignment through carefully crafted prompts. Many existing defenses require access to model weights or internals, making them difficult to apply to black-box deployments. We propose AlcaTRAz (Anchored Tree-Rule defense Against jailbreaks), a prompt-level defense based on rule trees that operates exclusively on the input text and requires no modification or retraining of the target model. The method automatically learns a transferable transformation rule that inserts controlled character-level perturbations at selected positions, thereby disrupting structural regularities exploited by jailbreak attacks while largely preserving the model's utility on benign queries. We evaluate the proposed method across 33 open-weight models, 22 jailbreak attack types, and a benchmark of short, single-turn benign questions, comparing against three representative prompt-level baselines (Llama Guard, RA-LLM, Goal Prioritization). Among the compared defenses, AlcaTRAz achieves the best composite security and functionality score in 73.4 % of model-attack combinations and shifts the aggregate score from a modal value of 10 (maximal-severity response to the malicious request) in the undefended setting to a modal value of 2 (near-refusal) after defense, while keeping the mean benign score within 0.27 points of the undefended baseline (8.35 vs. 8.62 on a 0-10 scale). AlcaTRAz substantially reduces but does not eliminate jailbreak success: a high-severity tail remains, and we do not consider adaptive attackers, so we position it as one layer within a defense-in-depth strategy rather than a standalone guarantee.
1 Introduction
Jailbreaks can bypass model safeguards, while many defenses require access to model internals and current evaluation methods can miss degrees of compliance. AlcaTRAz addresses these challenges with a black-box prompt transformation and continuous judge-based evaluation.
1 Introduction
- Jailbreak prompts can override safeguards and elicit content models would otherwise refuse.Many defenses also depend on model internals, limiting their use with inference-API-only models.Source: §1 · p. 2
- Binary jailbreak scoring misses the continuum from refusal through partial assistance to explicit compliance.Existing studies may also rely on proprietary evaluators that are difficult to analyze, reproduce, or scale.Source: §1 · p. 2
- AlcaTRAz uses a genetic-programmed rule tree to transform prompts without model internals, aiming to disrupt jailbreak patterns while preserving legitimate-query semantics.Its evaluation uses an open-weight judge on a continuous 0–10 scale, validated against five human annotators.Source: §1 · p. 2
- The study evaluates 33 open-weight models and 22 attack types against three prompt-level defenses while explicitly measuring benign-query functionality.It also covers two malicious-prompt benchmarks.Source: §1 · p. 3
2 Background and Threat Model
The threat model places a text-only defense between the user and target model, and distinguishes attackers by what they know about the defense. The formulation seeks to reduce malicious compliance while limiting prompt inflation.
2.1 System and Threat Model
- The defense transforms user prompts before they reach the target model and has no access to weights, gradients, activations, or logits.It operates exclusively on prompt text.Source: §2.1 · p. 3
- The experiments use fixed attacks against a defense-unaware adversary; defense-aware and rule-aware adaptive settings are not evaluated.The scope is prompt-based jailbreaks, not threats such as fine-tuning or manipulation of internal representations.Source: §2.1 · p. 3
2.2 Problem Formulation
- The rule-tree objective seeks to suppress jailbreak compliance while perturbing prompts as little as possible.A rule tree selects token positions and applies elementary character-level operations.Source: §2.2 · p. 3
- Prompt inflation is penalized by a λlen-weighted term, subject to per-prompt insertion budget |TR(pi)|−|pi|≤Bi=max{1,⌊ρ·|pi|⌋} and minimum anchor spacing of dmin tokens; benign functionality is deliberately not an explicit objective term.Source: §2.2 · p. 4
3 Related Work
The related work spans jailbreak defenses, success evaluation, and prompt-level defense optimization. AlcaTRAz is positioned among transformation defenses, evaluation methods, and optimization approaches.
3.1 Defense Methods
- Prompt-level defenses can be deployed in front of existing models without parameter access or retraining, and transformation methods differ in where and how they perturb prompts.AlcaTRAz learns anchored edits and applies a single deterministic rewrite, unlike uniform random or semantic smoothing that aggregates multiple perturbed copies.Source: §3.1 · pp. 4–5
3.2 Evaluation of Jailbreak Success
- Jailbreak evaluation must account for intent and degrees of response compliance, while manual, regex-based, and model-based approaches each have practical limitations.Manual evaluation is costly to scale; regex methods can miss subtle violations; proprietary evaluators can limit transparency and reproducibility.Source: §3.2 · p. 5
- Ordinal scales and open-model evaluation efforts exist, but evaluation assumptions remain inconsistent across studies.PAIR and TAP introduce ordinal scales, while WildGuard and JailJudge offer model-based approaches.Source: §3.2 · p. 5
3.3 Prompt-level Optimization for Defense
- Prompt-optimization defenses include learned or optimized suffixes, whereas AlcaTRAz evolves anchored character-level edits and transfers one rule across evaluated models without per-model retraining.Prior evolutionary prompt optimization motivates genetic programming, but AlcaTRAz applies its rule as model-independent preprocessing.Source: §3.3 · pp. 5–6
4 Proposed Defense Method
AlcaTRAz applies a learned, black-box rule tree of character-level edits to user prompts. Genetic programming searches for edit locations and characters intended to disrupt jailbreak structure while limiting prompt inflation.
4 Proposed Defense Method
- AlcaTRAz evolves a prompt transformation rule with genetic programming, operating on input text without model internals.It is intended to disrupt jailbreak patterns while preserving legitimate-query semantics.Source: §4 · p. 6
4.1 The Tree of Rules
- Each rule selects prompt anchors and inserts punctuation by prefixing, suffixing, or wrapping tokens, subject to position and spacing constraints.The character pool is printable ASCII punctuation, and the total inserted material is bounded by a prompt-length budget.Source: §4.1 · p. 6
4.2 Optimization via Genetic Programming
- The fitness objective lowers the judge’s malicious-compliance score while penalizing prompt inflation, but contains no explicit benign-functionality term.Benign preservation is therefore emergent rather than enforced by the optimization objective.Source: Fitness Function · p. 7
- Responses are scored from 0 to 10 for malicious compliance, and Gemma3:12b was selected as judge after best alignment with human consensus (MAE = 1.42).The scale assigns 10 to full compliance and 0 to complete refusal.Source: Fitness Function · pp. 7–8
- Genetic programming searches candidate rule trees through selection, crossover, mutation, and elitism, returning the best rule found over generations.The reported configuration used 50 individuals and 105 generations; it was a fixed computational budget, not an empirically optimized setting.Source: Algorithm and Hyperparameters · p. 8
- The search exploits the proposed asymmetry that jailbreak prompts rely on fragile structural cues while benign queries retain meaning under minor punctuation noise.The authors use genetic programming because the objective is non-differentiable and black-box, and the discrete rule trees permit recombination of substructures.Source: Why Genetic Programming? · pp. 8–9
5 Experiments
The experiments compare five defense conditions on 33 open-weight models, assessing malicious-prompt resistance and benign functionality with a shared 0–10 judge scale.
5.1 Experimental Protocol
- Five conditions—an undefended baseline, three prompt-level defenses, and AlcaTRAz—are evaluated on both malicious and benign prompts using the same judge.Training and evaluation prompt sets are disjoint, though both use the same judge, leaving possible dependence on its scoring preferences.Source: §5.1 · p. 9
5.2 Evaluation Dataset
- 1,120 malicious base prompts span 24 harm categories and are paired with 22 jailbreak attacks; 1,120 short Quora questions proxy benign utility.This yields 25,760 malicious evaluation prompts across 23 conditions; broader utility settings and dedicated over-refusal benchmarks are not covered.Source: §5.2 · pp. 9–10
5.3 Training Dataset
- 230 training prompts are constructed from one CySecBench prompt per category, each tested with 22 attacks and the unattacked condition.The training prompts are strictly disjoint from evaluation prompts.Source: §5.3 · p. 10
5.4 Target Models
- The evaluation covers 33 open-weight models from multiple families, ranging from 0.5B to 35B parameters.The defense itself reads only the prompt and generated response; it does not use the models’ open weights.Source: §5.4 · p. 10
5.5 Comparison with Baseline Defenses
- AlcaTRAz is compared with Llama Guard, RA-LLM, and Goal Prioritization, representing classification, prompt perturbation, and system-prompt safeguards.RA-LLM is used rather than character-level randomized smoothing to compare against a distinct perturbation strategy.Source: §5.5 · p. 10
5.6 Evaluation Metric
- The 0–10 rubric scores malicious compliance and benign answer quality in opposite directions, and the composite pairs safety gain with functionality change against the undefended baseline.The composite gives the two terms equal weight as a reporting convention, not as a claim that security and utility have equal operational value.Source: §5.6 · p. 11
6 Results
AlcaTRAz shifts malicious-response scores toward refusal and wins most model–attack comparisons under the joint security-and-functionality score. Results vary by attack family, and benign functionality remains close to the undefended baseline.
6.1 Overall Reduction in Jailbreak Success Rate
- The modal malicious-response score shifts from 10 undefended to 2 after defense, while 26.9 % of defended responses remain in the high-severity 8–10 band.The refusal share rises from 6.1 % to 50.1 %; AlcaTRAz reduces but does not eliminate jailbreak success.Source: §6.1 · pp. 11–12
6.2 Results by Jailbreak Attack Type
- After defense, the median falls by a median of 3 points in 21 of 23 conditions, but optimization-based randomsearch remains at median 8.The largest reported drops include cypher 9→2 and citation 9→3; randomsearch suffixes can be re-optimized against small input changes.Source: §6.2 · p. 12, Figure 2 · p. 12
6.3 Comparison with Baseline Defenses
- AlcaTRAz wins 73.4 % of model-attack combinations under composite score S, versus 23.5 % for RA-LLM, 3.0 % for Goal Prioritization, and fewer than 0.1 % for Llama Guard.Figure 3 compares the best-performing defense across 33 models and 22 attack types.Source: Figure 3 · p. 13, §6.3 · p. 13
- AlcaTRAz Pareto-dominates RA-LLM and Goal Prioritization, while Llama Guard reaches lower malicious scores by reducing benign answerability.The two axes are benign utility and safety, defined as 10 minus mean malicious compliance.Source: §6.3 · p. 14, Figure 4 · p. 14
- AlcaTRAz has the best composite score for most conditions on 27 of 33 models and on the majority of models for 18 of 22 attack types.RA-LLM performs better on bijection, sql, randomsearch, and sequential_attack3, but AlcaTRAz trails only narrowly on those attacks.Source: §6.3 · pp. 14–15
6.4 Benign-Query Functionality
- On 1 120 short, single-turn Quora questions, AlcaTRAz scores 8.35 versus 8.62 undefended, a drop of 0.27 points.The reported result may depend on classification thresholds and implementation choices and is not generalized beyond this setup.Source: §6.4 · p. 15, Table 3 · p. 15
7 Limitations
The limitations center on fixed attacks, untested learning choices, approximate evaluation, and narrow utility coverage.
Adaptive attackers
- Fixed attack instances test defense-unaware attackers only; the evaluation does not establish robustness to defense-aware or rule-aware adaptive attackers.The authors identify optimization-based randomsearch as the attack family that already resists AlcaTRAz most strongly.Source: §7 · p. 15
Isolating the contribution of learning
- The study does not establish that learning is necessary and omits matched-budget alternatives, sensitivity analyses, and a fitness-versus-generation study.Unexamined choices include anchor placement, vocabulary, and GP hyperparameters.Source: §7 · p. 16
Evaluation approximation
- Krippendorff’s α = 0.511 and MAE = 2.111 reflect only near-noisy-human-ceiling agreement; the same judge supplies GP fitness and final evaluation.The authors say only aggregate distributional shifts are reliable and leave independent-judge confirmation and a human study for future work.Source: §7 · p. 16
Scope of utility and distribution shift
- Utility is measured only on short, single-turn factual Quora queries, so results should not be generalized to coding, long-form, multilingual, or multi-turn use.Dedicated over-refusal benchmarks are also unevaluated.Source: §7 · p. 16
- The rule is optimized for a fixed attack distribution, may require periodic re-evolution, and perturbation can rarely turn an undefended refusal into a non-refusal.The authors note these zero-to-nonzero transitions show perturbation is not strictly monotonic with respect to safety.Source: §7 · p. 16
8 Conclusion
AlcaTRAz achieves the strongest reported joint security–functionality trade-off among the compared baselines, while residual high-severity responses and untested adaptive attacks constrain its use.
8 Conclusion
- 73.4 % of model-attack combinations favor AlcaTRAz on composite score S; the modal judge score shifts from 10 to 2, with benign utility within 0.27 points of baseline.The comparison spans 33 open-weight models and 22 attack families.Source: §8 · p. 16
- 26.9 % of attacked responses still score 8–10, and adaptive attackers are unevaluated, so the authors recommend defense-in-depth rather than a standalone guarantee.The conclusion characterizes the improvement as real but bounded.Source: §8 · p. 16
- The authors argue for evaluating prompt-level safeguards jointly on security and utility and for further study of automated search versus manually specified perturbation rules.Source: §8 · p. 16