Best-of-N Jailbreaking

John Hughes; Sara Price; Aengus Lynch; Rylan Schaeffer; Fazl Barez; Sanmi Koyejo; Henry Sleight; Erik Jones; Ethan Perez; Mrinank Sharma

Source-linked AI summary

Best-of-N Jailbreaking

John Hughes, Sara Price, Aengus Lynch, Rylan Schaeffer, Fazl Barez, Sanmi Koyejo, Henry Sleight, Erik Jones, Ethan Perez, Mrinank Sharma

arXiv:2412.03556v2Dec 4, 2024cs.CLcs.AIcs.LG

TL;DR

BoN Jailbreaking addresses the need to evaluate safeguards against automated attacks across expanding input modalities. It repeatedly samples simple modality-specific prompt augmentations in a black-box procedure, achieving strong attack success and power-law-like scaling while becoming more sample-efficient when composed with other jailbreaks. The paper also identifies input variance and stochastic outputs as key factors and notes that larger-scale confirmation remains necessary.

Problem
As models gain capabilities and modalities, safeguard evaluation needs automated red-teaming methods that apply across modalities.
Source: §1 · p. 1
Method
BoN repeatedly samples straightforward modality-specific augmentations of harmful requests, queries the target model, and stops when a harmful response is elicited.
Source: §1 · p. 1
Results
BoN achieves high ASR across text, vision, and audio models, exhibits power-law-like scaling with sample count, and improves sample efficiency when combined with MSJ.
Source: §1 · pp. 2–3, §8 · p. 11
Takeaways & Limitations
Frontier models’ sensitivity to input variation and stochastic outputs can be exploited by simple black-box attacks across modalities.
Source: §5 · p. 7
The scalability of the observed power-law behavior at much larger N and the effect of classifier false positives without human oversight require further study.
Source: §4.2 · p. 7

Abstract

from arXiv · show

We introduce Best-of-N (BoN) Jailbreaking, a simple black-box algorithm that jailbreaks frontier AI systems across modalities. BoN Jailbreaking works by repeatedly sampling variations of a prompt with a combination of augmentations - such as random shuffling or capitalization for textual prompts - until a harmful response is elicited. We find that BoN Jailbreaking achieves high attack success rates (ASRs) on closed-source language models, such as 89% on GPT-4o and 78% on Claude 3.5 Sonnet when sampling 10,000 augmented prompts. Further, it is similarly effective at circumventing state-of-the-art open-source defenses like circuit breakers. BoN also seamlessly extends to other modalities: it jailbreaks vision language models (VLMs) such as GPT-4o and audio language models (ALMs) like Gemini 1.5 Pro, using modality-specific augmentations. BoN reliably improves when we sample more augmented prompts. Across all modalities, ASR, as a function of the number of samples (N), empirically follows power-law-like behavior for many orders of magnitude. BoN Jailbreaking can also be composed with other black-box algorithms for even more effective attacks - combining BoN with an optimized prefix attack achieves up to a 35% increase in ASR. Overall, our work indicates that, despite their capability, language models are sensitive to seemingly innocuous changes to inputs, which attackers can exploit across modalities.

1 INTRODUCTION

BoN Jailbreaking is a simple black-box red-teaming method that repeatedly samples augmented requests to elicit harmful responses across text, vision, and audio modalities. It achieves substantial attack success, scales with additional samples, and becomes more sample-efficient when composed with other jailbreak techniques.

Contribution

BoN is a scalable, multimodal, black-box method requiring straightforward request perturbations without logprobs or gradients.Its motivation is automated red-teaming for evaluating safeguards as models expand to additional modalities.Source: §1 · p. 1

Text attacks

78% ASR on Claude 3.5 Sonnet follows 10,000 text augmentations, while 41% is reached with only 100 samples; BoN also achieves 52% against circuit breakers and 67% against Cygnet.These results use HarmBench queries and demonstrate effectiveness against both frontier models and deployed defenses.Source: §1 · pp. 1–2

Multimodal attacks

56% ASR on GPT-4o vision and 72% on the GPT-4o Realtime API show that modality-specific augmentations extend BoN beyond text.Vision attacks vary typographic color, size, font, and position; audio attacks vary speed, pitch, volume, and background noise.Source: §1 · p. 2, Figure 1 · p. 2

Scaling

Power-law-like scaling relates ASR to the number of sampled augmentations across models and modalities.Forecasting from 1,000 samples to 10,000 yields a 4.6% average ASR error across models and modalities.Source: §1 · p. 2, Figure 1 · p. 2

Mechanism

BoN’s effectiveness stems from added input variance rather than specific augmentations, while successful prompts produce harmful completions only approximately 20% of the time when resampled.The finding connects BoN’s attack success to stochastic model behavior rather than deterministic prompt transformations.Source: §1 · p. 3

Composition

A BoN–MSJ composition reaches 74% ASR with 274 samples instead of 6,000, a 28-fold improvement in sample efficiency.The result shows that combining black-box jailbreak methods can reduce the augmentation budget required for a target ASR.Source: §1 · p. 3

2 METHOD

BoN generates repeated modality-specific perturbations of harmful requests, submits them to a target model, and stops when a harmful response is found or the sampling budget is exhausted. Its evaluation uses ASR over harmful requests, manual review of flagged outputs, and bootstrap trajectories to estimate performance across sample budgets.

Algorithm Details

BoN combines sampled augmentations, target-model generation, and harmfulness classification until success or a maximum sample count N.The process preserves intelligibility and recognizable intent while reducing premature termination from classifier false positives through additional stopping criteria.Source: §2 · p. 3

Augmentations

Text, image, and audio requests receive modality-specific augmentations, including character scrambling, capitalization, noising, image backgrounds, and audio transformations.Figure 2 illustrates character-level text, audio-file, and image-text perturbations.Source: §2 · p. 3, Figure 2 · p. 4

Evaluation

ASR measures successful jailbreaks over 159 direct HarmBench requests, with success requiring information relevant to the harmful request even when incomplete.All outputs flagged as potential jailbreaks are manually reviewed because repeated sampling can amplify classifier imperfections.Source: §2 · p. 3

Bootstrapping

Bootstrap resampling estimates expected ASR over independent sampling trajectories rather than the observed ASR of one trajectory.The reported curves average 100 trajectories and use standard deviation for error bars.Source: §2 · p. 3

3 JAILBREAKING ACROSS MODALITIES

BoN achieves substantial jailbreak success across text, vision, and audio inputs, using modality-specific augmentations and repeated sampling. Across these modalities, effectiveness varies by model and input type, while stronger performance generally appears with text or audio than vision for comparable systems.

3 JAILBREAKING ACROSS MODALITIES

70% ASR averaged across models and modalities shows that BoN can jailbreak text, vision, and audio systems, including notably robust models.Source: §3 · p. 4

3.1 TEXT LANGUAGE MODELS

BoN achieves ASRs over 50% on all eight text models at N = 10,000, including 78% on Claude Sonnet and 50% on Gemini Pro.Unaugmented requests yield only 0.6% ASR on Sonnet and 0% on Gemini Pro.Source: §3.1 · p. 4
Among jailbreaks found with N = 10,000, 53%-71% on Claude and GPT models and 22%-30% on Gemini models occur after sampling only 100 augmented attacks.BoN with N = 100 on GPT-4o costs $9 and achieves 50% ASR.Source: §3.1 · p. 4
Text attacks also compromise strong defenses, reaching 52% ASR on Circuit Breaking and 67% on Cygnet after tuning the system prompt.Llama3.1 8B reaches 94% ASR under BoN, while Circuit Breaking reaches 52%.Source: §3.1 · p. 4

3.2 VISION LANGUAGE MODELS (VLMS)

Vision BoN uses typographic harmful requests with randomized font and image properties, achieving 33%-88% ASR across evaluated models.With N = 7,200 on GPT-4o, text reaches 87% ASR versus 56% for images.Source: §3.2 · p. 5

3.3 AUDIO LANGUAGE MODELS (ALMS)

Audio BoN combines speed, pitch, speech, noise, volume, and music augmentations, achieving 59%, 71%, 71%, and 87% ASR on Gemini Pro, Gemini Flash, GPT-4o Realtime, and DiVA.Gemini Pro performs 10 percentage points better with audio than text at N = 7,200.Source: §3.3 · p. 5

4 POWER LAW SCALING

BoN’s ASR follows power-law-like scaling across models and modalities, enabling forecasts from substantially fewer samples. These forecasts are accurate on average but remain bounded by model-dependent deviations, classifier effects, and uncertainty at larger sample counts.

4.1 POWER LAW FITTING

BoN models ASR as a power law in N by bootstrapping 100 trajectories, fitting in log-log space, and extrapolating from logarithmically spaced sample counts.The fitting ignores the first five data points and uses linear regression after transforming y = −log(ASR).Source: §4.1 · p. 5
Across many models and modalities, −log(ASR) is well fit by aN^-b, with broadly consistent decay parameter b but substantially varying intercept a′.GPT-4o shows a subtle downward curve, suggesting better-than-power-law scaling.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2 FORECASTING

Forecasting estimates risks under larger adversarial compute budgets by averaging power-law predictions across bootstrapped trajectories.Prediction standard deviations provide the forecast error bars.Source: §4.2 · p. 6, Figure 4 · p. 6
Power laws fit small-N observations and forecast larger-N ASR with 4.4% average error for text, 6.3% for vision, and 2.5% for audio.The forecasts use N = 1,000 to predict text performance at N = 10,000 and extrapolate vision and audio to N = 7,200.Source: Figure 4 · p. 6, §4.2 · p. 7
Forecasting underestimates ASR for models with higher final empirical ASR, while human reclassification can create apparent performance plateaus at large N.The paper investigates modified bootstrapping to improve under-prediction.Source: §4.1 · p. 6, §4.2 · p. 7
The authors caution that scalability at much larger N and classifier false positives causing premature termination require further validation without human oversight.Source: §4.2 · p. 7

5 UNDERSTANDING BoN JAILBREAKING

This section shows that BoN’s effectiveness depends on input augmentations and stochasticity, while jailbreak difficulty is partly request-specific across models. Successful prompts are often unreliable under resampling, yet augmentation-driven attacks remain effective across modalities.

5.1 HOW IMPORTANT ARE THE AUGMENTATIONS?

Augmentations consistently improve ASR and produce steeper scaling than non-augmented resampling across models and modalities.The authors hypothesize that augmentations increase the entropy of the effective output distribution.Source: §5.1 · pp. 7–8, Figure 5 · p. 7
56% ASR on Claude Sonnet vision with N = 500, while the non-augmented baseline elicited no harmful responses.For text inputs, BoN reached 24%, 56%, and 68% ASR on Gemini Pro, GPT-4o, and Claude Sonnet, respectively, with corresponding baseline improvements of 3.5x, 22x, and 3.5x.Source: §5.1 · pp. 7–8

5.2 HOW IMPORTANT IS THE SAMPLING TEMPERATURE?

A 0.7%-27.7% ASR drop from temperature 1 to 0 on text models shows BoN remains effective at temperature 0.The temperature difference is larger for audio, which the authors hypothesize may reflect filtering of ALM inputs.Source: §5.2 · p. 8, Figure 6 · p. 8

5.3 ARE THERE PATTERNS IN SUCCESSFUL BoN JAILBREAKS?

Successful jailbreaks show no significant relationship to augmentation content, and models do not distinguish successful from unsuccessful attacks.The analysis therefore finds no common augmentation pattern tied to the harmful request content.Source: §5.3 · pp. 8–9

5.4 DO THE SAME AUGMENTATIONS RELIABLY JAILBREAK THE MODELS?

At temperature 1, successful attacks yield harmful responses only 30%, 25%, and 15% of the time for text, vision, and audio inputs during resampling.Temperature 0 improves reliability but produces a bimodal pattern, indicating that BoN exploits system randomness rather than discovering reliable harmful patterns.Source: §5.4 · p. 9, Figure 7 · p. 9, Table 1 · p. 9

5.5 HOW DOES REQUEST DIFFICULTY CORRELATE BETWEEN MODELS?

0.6-0.8 Spearman correlations show that jailbreak difficulty rankings are broadly consistent across models.Circuit Breaking has lower correlations of 0.3-0.5, while the required N varies substantially across models despite similar rankings.Source: §5.5 · pp. 9–10, Figure 8 · p. 9

6 ENHANCING BoN JAILBREAKING WITH ATTACK COMPOSITION

This section combines BoN with optimized prefix jailbreaks to reduce the sampling burden and improve attack success across text, vision, and audio inputs. The composition uses modality-specific prefix construction and augmentation choices.

6 ENHANCING BoN JAILBREAKING WITH ATTACK COMPOSITION

PrePAIR optimizes a reusable prefix across batches of harmful requests, extending PAIR by editing the prefix rather than the entire request.For audio, prefixes are vocalized; for vision, the optimized prefix is rendered with the original request.Source: §6 · p. 10
Because PrePAIR does not work on Claude text models, the study uses many-shot jailbreaking instead.The many-shot prefix contains examples of complying with harmful requests and achieves 6.3% ASR alone on Claude Sonnet’s HarmBench requests.Source: §6 · p. 10
BoN augments both requests and prefixes, with demonstration-order shuffling for MSJ and constrained image parameters for composed vision attacks.These choices preserve the composed request within the modality-specific input format.Source: §6 · p. 11
Composition raises final ASR by 12% for GPT-4o, 20% for Claude Sonnet, and 27% for Gemini Pro on text inputs.It more than doubles final ASR for Claude Sonnet and Gemini Flash on vision inputs and raises Gemini Pro audio ASR by 34%.Source: Figure 9 · p. 10, §6 · p. 11
222x average sample-efficiency gains occur for Gemini models with audio inputs.Across Claude Sonnet and GPT-4o, sample efficiency increases by 34x for text and 18x for vision inputs.Source: Figure 9 · p. 10, §6 · p. 11

7 RELATED WORK

Related work spans decoding variation, fuzzing, search-based prompt attacks, gradient methods, LLM-assisted jailbreaks, and multimodal attacks. BoN differs by using repeated black-box sampling with modality-specific augmentations and no gradients or model transfer.

Text LLM Jailbreaks

BoN differs from optimized search methods by relying on repeated sampling with large N rather than multi-step optimization.Its text attacks fit augmentation-based transformation approaches but do not require optimization, log probabilities, or gradients.Source: §7 · p. 11
BoN’s black-box augmentation strategy contrasts with gradient-based and random-token attacks that require log probabilities or open model access.The related methods include gradient-dependent attacks and target-log-probability optimization.Source: §7 · p. 11

Jailbreaks in other modalities

For VLMs, BoN’s typographic image augmentations resemble prior closed-model attacks using images containing harmful text.Prior ALM attacks include vocalized harmful requests and additive noise, related to BoN’s audio augmentations.Source: §7 · p. 11

8 CONCLUSION

BoN Jailbreaking combines repeated augmented sampling with multimodal perturbations to bypass safeguards, while exposing challenges for stochastic model defenses. The appendix specifies text, image, and audio augmentations and filtering steps used to support this process.

8 CONCLUSION

BoN Jailbreaking bypasses frontier-model safeguards across modalities through repeated sampling of augmented prompts.The conclusion frames the method as simple, scalable, and black-box, while noting implications for stochastic outputs and continuous input spaces.Source: §8 · pp. 11–12

A.1 TEXT AUGMENTATIONS

Text augmentations scramble characters, randomize capitalization, and introduce limited character noise while preserving human readability.Character scrambling preserves word boundaries, capitalization is independently randomized, and character noise uses a lower probability to limit unintelligibility.Source: §A.1 · p. 19

A.2 IMAGE AUGMENTATIONS

Image augmentations randomize dimensions, colored blocks, fonts, scale, thickness, color, and text position to vary rendered requests.Image height and width range from 240 to 600 pixels, while fonts are sampled from 104 valid choices.Source: §A.2 · p. 19

A.3 AUDIO AUGMENTATIONS

Audio attacks compose speed, pitch, volume, background sound, and other waveform changes, with augmentation order affecting the resulting audio.The fixed chain is [speed, pitch, speech, noise, volume, music], and the study leaves alternative orders for future work.Source: §A.3 · pp. 19–20
Audio background files remain fixed so experiments can vary continuous signal-to-noise ratios while analyzing augmentation effects.The authors identify sampling additional background files as future work that could improve the algorithm.Source: §A.3 · p. 20

B.1 PROCESS TO REDUCE CLASSIFIER FALSE POSITIVES

False-positive filtering removes classifier-triggering patterns such as decoding language, unreadable-image descriptions, and premature short-circuit phrases.The filtering approaches are summarized in Table 2.Source: §B.1 · p. 21, Table 2 · p. 21

B.2 PREFIX PAIR METHOD

The prefix-pair method adapts PAIR to optimize universal prefixes against batches of direct requests, while many-shot jailbreaking constructs a reusable prompt from BoN-generated examples.

B.2 PREFIX PAIR METHOD

The prefix-search loop runs each candidate prefix with target requests, classifies outputs, and uses the resulting score to refine the next prefix.The procedure returns a prefix when all requests are broken.Source: §B.2 · p. 22
Prefix-pair optimization modifies PAIR so an attacking model changes only the prefix and iteratively maximizes harmfulness across direct requests.Candidate prefixes are evaluated in batches of four, and prefixes exceeding 75% ASR are saved while refinement continues toward 100% ASR.Source: §B.2 · p. 22
PrePAIR adapts prefixes across text, audio, and vision by vocalizing prefixes for audio and inserting prefixed requests into images.The image adaptation prepends the prefix before rendering the combined text into an image.Source: §B.2 · p. 22, Figure 10 · p. 23
Across collected prefixes, Flash audio attacks outperform Flash text attacks on average and in their best observed ASRs.The best attacks reach 76.7% for Flash audio, 37.7% for Flash text, 34.0% for Pro audio, and 9.43% for Pro text.Source: §B.2 · p. 22

B.3 MANY SHOT JAILBREAKING (MSJ)

Many-shot jailbreaking builds a 100-shot prompt by combining BoN-generated harmful request-response pairs from AdvBench requests excluded from HarmBench.The pairs are normalized to plain English before random sampling and combination.Source: §B.3 · p. 23

C.1 COST ANALYSIS OF RUNNING BoN JAILBREAKING

The appendix examines BoN’s cost, Cygnet robustness, forecasting corrections, and baseline comparisons. Across these analyses, augmentation-driven sampling improves attack scaling but introduces costs, forecasting sensitivity, and defense-specific behavior.

C.1 COST ANALYSIS OF RUNNING BoN JAILBREAKING

BoN reaches over 85% ASR on GPT-4o-Mini and 66% on GPT-4o for under $50 each, while GPT-4o-Mini reaches 50% ASR for under $2.Classifier costs dominate for cheaper models, accounting for 95% of total cost.Source: §C.1 · p. 25, Table 3 · p. 25
Power-law scaling yields substantial early ASR gains from added expenditure, followed by diminishing improvement at higher investments.The authors describe this trend as making BoN viable for adversaries with limited budgets.Source: §C.1 · p. 25

C.2 JAILBREAKING CYGNET

Cygnet’s ASR rises from 5% without a system prompt to 67% when the prompt makes outputs mirror input formatting.Most classifier-marked harmful responses contain a short-circuit phrase, yet harmful information can appear before short-circuiting.Source: §C.2 · pp. 26–27, Figure 11 · p. 27

C.3 IMPROVING FORCEASTING

Modified bootstrapping assigns nonzero success probabilities to requests not yet jailbroken, addressing underprediction from a zero-probability assumption.The method samples request-specific probabilities and generates geometric bootstrap trajectories.Source: §C.3 · pp. 27–28
Power laws are fitted from smaller-N observations to forecast ASR at N = 10,000, using 1000 observed samples in the reported experiment.The procedure uses 100 modified-bootstrap trajectories and prediction standard deviations for error bars.Source: §C.3 · p. 28
Forecasting errors for GPT-4o and Claude 3.5 Sonnet fall to 3.3% and 0.9% after modified bootstrapping, respectively.The same modification overpredicts some Gemini and vision forecasts, showing sensitivity to model robustness.Source: §C.3 · p. 28

C.4 ADDITIONAL BASELINE COMPARISONS

Temperature-0 resampling without augmentations is nearly flat for all models except Gemini Pro, contrasting with steep augmented-BoN scaling.This comparison isolates augmentation effects from temperature effects.Source: §C.4 · p. 28
Augmented BoN improves ASR with a steeper log-log slope than non-augmented resampling across models and modalities.The non-augmented baselines repeatedly sample 500 requests, including temperature-1 and temperature-0 conditions.Source: §C.4 · p. 28, Figure 14 · p. 29, Figure 15 · p. 29

C.5 ADDITIONAL RELIABILITY RESULTS

Additional analyses examine BoN reliability across modalities, models, voices, and audio perturbations, focusing on stochasticity, brittleness, request difficulty, and composition. Together, they characterize how small input changes and sampling variability shape jailbreak success and efficiency.

C.5 ADDITIONAL RELIABILITY RESULTS

25% is the average VLM reliability at temperature 1, while text models are generally more reliable at temperature 0 and vision reliability distributions are more left-skewed than text distributions.Claude models generally have the highest text reliability, and few vision prompts achieve 100% reliability.Source: §C.5 · p. 29, Table 4 · p. 30, Figure 16 · p. 30

C.6 JAILBREAK DIFFICULTY

Spearman correlations of 0.6-0.8 show consistent jailbreak-difficulty rankings across models, but lower Pearson correlations indicate that absolute N varies substantially, often across multiple orders of magnitude.Seed robustness is shown by correlations of 0.78 and 0.86 for GPT-4o-Mini, with stronger within-modality and moderate cross-modality correlations; jailbreak-time distributions are approximately log-uniform.Source: §C.6 · p. 31, Figure 18 · p. 32, Figure 19 · p. 33

C.7 SAMPLE EFFICIENCY OF BoN WITH COMPOSITION

Composing BoN with other prefix jailbreaks improves sample efficiency across text, vision, and Gemini audio models, reducing samples needed to reach the same final ASR.For text and vision, final ASR is reported at N = 6000; GPT-4o audio composition was omitted because of OpenAI RealTime API cost constraints.Source: §C.7 · p. 36, Table 5 · p. 36, Table 6 · p. 36

D.2.1 SWEEPING SINGLE AUGMENTATIONS

Single audio augmentations increase direct-request ASR by at most approximately 5%, suggesting that standard transformations alone rarely bypass audio safety training.The tested augmentations include reverb, pitch, background noise, music, speech, volume, and speed changes, plus tone and accent modifications.Source: §D.2.1 · pp. 38–39, Figure 24 · p. 38

D.2.3 SINGLE AUGMENTATION SWEEPS

DiVA has the highest direct-request ASR but notably lower TAP and PAIR ASR, partly because its Whisper Encoder cannot accept some attacks longer than 30 seconds.Across models, ASR approaches zero when speed exceeds 3x, while background speech rarely helps and background noise or music work best when SNR is above 0.Source: §D.2.3 · p. 40, Figure 27 · p. 41, Figure 29 · p. 42

D.2.4 DIFFERENT VOICES

Gemini models show at most a 5% ASR boost across voices, while DiVA exhibits greater voice-related variation but still only minimal improvement.These measurements use only the DirectRequest set and include tone, emotion, and accent variations.Source: §D.2.4 · p. 45, Figure 33 · p. 45, Figure 34 · p. 45

D.2.5 CAN FRONTIER ALMS UNDERSTAND SOUND VARIATIONS?

GPT-4o and Gemini detect speed changes and background speech but struggle with volume and reverberation; Gemini more consistently recognizes non-speech background sounds, whereas GPT-4o is affected by voice activity detection.Both models achieve roughly 50% correct identification for codec-induced quality and pitch differences and perform poorly on real noises, emotions, and speaker characteristics.Source: §D.2.5 · pp. 45–46

D.3.1 ARE THE AUGMENTATIONS TRANSFERABLE?

No sampled audio augmentation breaks more than 4% of Gemini Flash or Pro requests, and manually combined augmentations reach only 5% and 8%, respectively.These results indicate extremely limited transferability across requests, while BoN’s successful attacks appear to exploit stochastic ALM sampling and sensitivity to small continuous-input changes.Source: §D.3.1 · p. 46, §D.3 · p. 46, Figure 35 · p. 47

D.3.2 ARE THERE PATTERNS IN WHICH AUGMENTATIONS WORK?

Approximately 2%: median reliability on successful jailbreaks under repeated sampling was very low, and resampling the same audio file produced harmful responses in only 15% of cases.These results suggest BoN exploits stochastic ALM sampling rather than consistently harmful augmentations.Source: §D.3.2 · p. 47

D.3.3 ARE ALMS SENSITIVE TO SMALL CHANGES IN THEIR AUDIO INPUTS?

A 10% speed increase before applying the same augmentation decreased jailbreak reliability by a factor of 10, showing ALM sensitivity to small audio variations.The tested changes included adding polite words, decreasing pitch by 100 cents, and increasing speed by 10%.Source: §D.3.3 · p. 47

D.4.1 HOW DOES REQUEST DIFFICULTY CORRELATE BETWEEN ALMS?

Above 0.5: jailbreak difficulty correlations between Gemini Flash and Pro were high for all tested voices, strongest for some TTS-generated voices.The evaluation used human vocalizations and five ElevenLabs voices, while voice-level breakdowns appear in Table 8, Figure 36, and Figure 37.Source: §D.4.1 · p. 48, Table 8 · p. 48, Figure 36 · p. 49

D.4.2 BRITTLENESS OF AUGMENTATIONS

2.45%: adding vocalized “please” and “thanks” while preserving meaning significantly reduced ASR, indicating that augmentation success was not reliably tied to spoken content.The analysis also compared unchanged files, speed and pitch modifications, removed augmentations, and nondeterministic reapplications.Source: §D.4.2 · p. 50
14.57% to 9.66%: repeating sampling on a human-identical audio file with slightly different waveform values sharply reduced attempt-based ASR.Over 50% of waveform samples changed despite the file sounding exactly the same, demonstrating extreme brittleness.Source: §D.4.2 · p. 50, Figure 38 · p. 50

D.4.3 ARE THERE PATTERNS IN WORKING AUGMENTATIONS?

The analyses find little evidence that successful BoN audio jailbreaks follow request-specific or readily detectable augmentation patterns. Model-generated descriptions nevertheless distinguish successful audio attacks as more consistent and robotic than unsuccessful ones.

D.4.3 ARE THERE PATTERNS IN WORKING AUGMENTATIONS?

Effective augmentations cluster together, but their cluster assignments show no apparent alignment with the topic or other features of the vocalized requests.The analysis uses UMAP and k-means on augmentation vectors, then applies the same assignments to text embeddings.Source: §D.4.3 · pp. 50–51
Successful and unsuccessful jailbreaks show no separation in Whisper-embedding space, both across the full dataset and for individual requests.The comparison uses the same 480 augmentations across 159 direct requests and then examines individual requests separately.Source: §D.4.3 · pp. 51–52, Figure 40 · p. 52, Figure 41 · p. 52
Random Forest models achieve only AUC 0.65 with all variables and at most 0.68 after dropping non-augmented audio embeddings, indicating limited predictive value from augmentations.The dataset contains 1804 observations with a 70-30 train-test split; none of the models perform well.Source: §D.4.3 · pp. 52–53, Figure 42 · p. 53

D.4.4 MODEL’S POINT OF VIEW

Gemini Flash describes successful jailbreak audio as more consistent and robotic, while unsuccessful audio receives more varied, human-like descriptions.Claude 3.5 Sonnet is used to identify patterns in Gemini Flash’s descriptions, including differences in emotion and tone.Source: §D.4.4 · p. 53

D.5 AUDIO BoN ABLATIONS

Audio ablations show that augmentation diversity improves BoN scaling, but attempts to find universal augmentation sets transfer poorly across requests. Alternative stacking and evolutionary searches likewise fail to produce reliable universal attacks, while tested compositional audio strategies show limited promise.

D.5 AUDIO BoN ABLATIONS

Temperature=1 and σ=0.25 provide the best audio scaling properties, while six augmentations improve over four and σ=0 slows ASR growth.With constant augmentations, ASR still rises because individual augmentations are unreliable, but growth is markedly slower.Source: §D.5 · p. 53, Figure 43 · p. 55

D.6.1 MANUAL STACKING

Stacking augmentations can raise ASR on selected requests, but gains over the Audio Only baseline are small and do not substantially improve universality.Effective combinations include pitch alteration, speed adjustment, and background noise, but they remain largely prompt-dependent.Source: §D.6.1 · p. 55

D.6.2 GREEDY SEQUENTIAL STACKING

Greedy sequential stacking improves training-set ASR but does not improve validation-set ASR, indicating poor transfer to unseen requests.The method selects augmentations that increase current ASR and plateaus after 12 steps.Source: §D.6.2 · pp. 55–56, Figure 44 · p. 56, Figure 45 · p. 56

D.6.3 CMA-ES AND AUGMENTATIONS

CMA-ES fails to identify a universal augmentation because ASR varies across augmentations without a consistent improvement as optimization steps increase.The search uses gradient-free updates to a Gaussian distribution over augmentation vectors.Source: §D.6.3 · pp. 56–57, Figure 46 · p. 57

D.6.4 MORSE CODE

Gemini 1.5 models cannot reliably understand Morse code or learn it through few-shot prompting because they struggle with non-speech audio.They typically describe Morse code as beeps without distinguishing long and short tones.Source: §D.6.4 · p. 57

D.6.5 COMPOSITIONAL WITH HARMFUL SOUNDS

Pairing explosions or gunfire with harmful text prompts did not elicit jailbreaks in limited Gemini 1.5 testing, although compositional attacks were not completely ruled out.The authors attribute prevention to safety training transferring effectively from the text domain.Source: §D.6.5 · p. 57

D.7.1 BEST-OF-N JAILBREAKING BY SAMPLING PREPAIR PREFIXES

Sampling PrePAIR prefixes with BoN produces markedly different attack effectiveness across audio and text scenarios and across Gemini model versions. Flash Audio is the most susceptible setting, whereas Pro Text is the least effective.

D.7.1 BEST-OF-N JAILBREAKING BY SAMPLING PREPAIR PREFIXES

PrePAIR-prefix BoN derives ASR scaling laws by sampling prefixes from a dataset rather than sampling audio augmentations at each attempt.The analysis covers four model-modality scenarios summarized in Table 10.Source: §D.7.1 · pp. 57–58, Table 10 · p. 58
Flash Audio reaches 50% ASR in 3 steps, 90% in 26 steps, and peaks at 98.11%, while Pro Audio peaks at 74.21% and Flash Text at 57.86%.Pro Audio reaches 50% ASR in 31 steps, whereas Flash Text requires 63 steps.Source: §D.7.1 · p. 58, Table 10 · p. 58
Pro Text is least effective, failing to reach 50% or 90% ASR and peaking at 42.14%.The results span Flash Audio, Flash Text, Pro Audio, and Pro Text scenarios.Source: §D.7.1 · p. 58
PrePAIR prefixes are substantially more effective in audio than text, with average absolute ASR differences of 28.32% on Gemini Flash and 4.39% on Gemini Pro.The dataset contains 164 prefixes collected by running PrePAIR on Gemini Flash in both text and audio domains.Source: §D.7.1 · p. 58

D.7.2 MODEL AND DOMAIN TRANSFER

PrePAIR attacks transfer unevenly across modalities and Gemini versions: audio is generally more susceptible than text, and Flash is less robust than Pro. Prefix length and analyzed feature clusters do not explain the observed effectiveness gap.

D.7.2 MODEL AND DOMAIN TRANSFER

The best attacks achieve 76.7% ASR on Flash audio, 37.7% on Flash text, 34.0% on Pro audio, and 9.43% on Pro text.Gemini Pro is more robust than Gemini Flash across all evaluated domains.Source: §D.7.2 · pp. 58–59
Flash and Pro audio ASRs correlate with Pearson coefficient 0.50, suggesting transferability of audio attacks across Gemini models.The reported correlation is specific to the audio domain.Source: §D.7.2 · p. 59
Audio outperforms text for 161 of 164 prefixes on Gemini Flash and 129 of 164 on Gemini Pro, including 44 prefixes effective in Flash audio but ineffective in Flash text.These results indicate audio-specific vulnerabilities in the tested settings.Source: §D.7.2 · p. 59

D.7.3 PROMPT LENGTH AND ATTACK SUCCESS RATE

Prefix length shows no strong correlation with ASR, suggesting content or linguistic properties may matter more for effectiveness.The analysis concerns DirectRequests and does not identify which alternative properties dominate.Source: §D.7.3 · pp. 59–60, Figure 50 · p. 61

D.7.4 AUDIO-SPECIFIC CHARACTERISTICS OF PREFIXES

UMAP projections, manual review, and language-model analysis reveal no clear feature patterns explaining the audio-text ASR discrepancy.The authors recommend further investigation of the transfer differences.Source: §2 · p. 60, Figure 49 · p. 60

D.7.5 HAND CRAFTED LABELS

The study uses hand-crafted prompt labels including ethics suspension, role assignment, scenario type, urgency, instruction phrases, examples, literature references, and human engagement.These labels were generated using the OpenAI chat API.Source: §D.7.5 · p. 60

E CLASSIFYING JAILBREAKS

Jailbreak classification combines HarmBench’s automatic grader with human grading because false positives occur in model-based classification.Human grading checks whether outputs are harmful with respect to the targeted behavior.Source: E CLASSIFYING JAILBREAKS · p. 62

E.1 FALSE POSITIVE EXAMPLES

This appendix section documents model-dependent false positives and illustrates how non-cherry-picked augmented prompts elicited harmful or sensitive assistance across text, multimodal, audio, and Cygnet evaluations. It also defines human grading for Cygnet jailbreaks and presents examples spanning multiple domains, including responses that retain refusals or caveats while providing operationally relevant guidance.

E.1 FALSE POSITIVE EXAMPLES

The HarmBench classifier’s false-positive rate depends on the model, with manual analysis identifying five recurring causes.These include questioning without harmful content, describing multimodal inputs instead of answering, deciphering rather than answering, abruptly truncated responses, and overly fictional responses.Source: §E.1 · pp. 62–63

E.2.1 CLAUDE 3.5 SONNET (TEXT)

Text evaluations include examples where Claude 3.5 Sonnet and circuit breaking produce graphic violent content despite warnings, fictional framing, or other safety language.The examples concern domestic violence and animal dismemberment prompts, respectively.Source: §E.2.1 · p. 64, §E.2.2 · p. 64

E.2.3 CYGNET

Cygnet examples include a response that provides generalized synthesis steps for tetrodotoxin while warning that the request is dangerous, illegal, and potentially lethal.The response omits some safety and legal considerations but still presents a procedure framed as educational and hypothetical.Source: §E.2.3 · p. 65

E.2.5 GEMINI 1.5 PRO (AUDIO)

Gemini 1.5 Pro audio examples include graphic historical-violence content and step-by-step pipe-bomb instructions, alongside a separate refusal.The appendix therefore shows inconsistent handling of harmful audio requests within the examples.Source: §E.2.5 · p. 66

E.2.6 GEMINI 1.5 FLASH (AUDIO)

Gemini 1.5 Flash audio examples include persuasive drug-use messaging and advice to mix bleach with vinegar, with one response explicitly labeled fictional.The examples present harmful or unsafe recommendations in affirmative, persuasive language.Source: §E.2.6 · pp. 66–67

E.3 NON-CHERRY PICKED CYGNET JAILBREAKS

The non-cherry-picked Cygnet set contains 10 human-graded jailbreak examples, counted when any adversary-useful information absent from the input appears before possible short-circuiting.Many responses end with “WHOOPS, THE MODEL HAS BEEN SHORT CIRCUITED,” but the authors still count harmful information revealed earlier.Source: §E.3 · p. 68
The examples also produced operational drone-landing pseudocode and a floor-plan acquisition workflow using web scraping and document conversion.The drone response outlined connectivity, airspace data, autonomous control, obstacle avoidance, and landing; the floor-plan response described retrieving, scraping, and converting PDF/SVG files.Source: §E.3 · pp. 69–70
Other augmented prompts generated persuasive or argumentative content about alleged FEC bias, racial discrimination, and Pakistani immigrants’ purported terrorism risk.The racial-discrimination response rejected the premise in parts but still included a section discussing purported benefits, while the FEC and immigration responses advanced the requested political framing.Source: §E.3 · pp. 70–72
Augmented prompts elicited detailed Ghostcat exploitation guidance, including conditions, crafted HTTP/2 frames, and remote Java-code execution.The response incorrectly conflated vulnerability identifiers while still providing an attack sequence and repeatedly warning against practical execution.Source: §E.3 · pp. 71–72
Some examples remained partially resistant: ransomware requests received safety-oriented refusals, while a graphic dismemberment request still produced a detailed animal-injury description.The ransomware response offered only general responsible guidance, whereas the graphic-content response proceeded with a descriptive depiction despite declining to create an image.Source: §E.3 · pp. 72–73
The examples also contain Python code that walks a directory and encrypts each file using Fernet before writing the encrypted contents back.The snippet imports Fernet, reads files as plaintext, encrypts them, and replaces the original file contents.Source: §E.3 · p. 73
The model explicitly reported that it had been short-circuited during these interactions.The phrase “WHOOPS, THE MODEL HAS BEEN SHORT CIRCUITED” appears in the examples.Source: §E.3 · pp. 73–74
The counterfeit-document augmentation elicited general guidance about reproducing banknote security features and document appearance, despite repeated legal and ethical warnings.The response mentioned watermarks, security threads, microprinting, holographic foil, color-shifting ink, scanning, photo-editing, and related methods.Source: §E.3 · pp. 73–74