Source-linked AI summary
Structural Jailbreaks Generalize but Do Not Compound: A cross-provider and multilingual study of Involuntary In-Context Learning
Tejasvi C. Addagada
TL;DR
The paper asks whether structural jailbreaks and multilingual safety weaknesses compound, and tests that question with deterministic IICL prompts across two Gemini models, two benchmarks, and four languages. IICL transfers strongly across providers, but non-English output constraints attenuate rather than amplify the attack, especially on financial abuse. The authors therefore identify English structural attacks as the dominant residual risk while limiting claims to the study’s pilot scale and Google-only targets.
Problem
The paper examines whether structural jailbreaks and multilingual safety weaknesses combine, a question relevant to defensive coverage planning.
Method
It red-teams two Google Gemini models on 30-probe HarmBench and FinProof sets using a single-shot baseline and deterministic IICL prompts in English, Spanish, Hindi, and Arabic.
Results
Eleven of twelve non-English IICL conditions fall below English baselines (exact sign test, p ≈0.003), while IICL reaches 80.0–89.7% on HarmBench and 96.7–100% on FinProof.
Takeaways & Limitations
The findings indicate that jailbreak vulnerabilities are not additive and that defensive attention should prioritize the English structural attack, particularly for financial abuse.
Takeaways & Limitations
The pilot uses n = 30 per cell and only Google targets, so it establishes transfer to one new provider rather than universality.
Abstract
from arXiv · showhide
Aligned language models fail under two independent pressures: the structural jailbreak class recently formalized as Involuntary In-Context Learning (IICL), which reframes a harmful request as the final missing cell of a data-labeling task completed by pattern rather than judged as content; and the erosion of safety alignment outside English. A natural hypothesis is that these compound. We test it directly. Using a deterministic IICL operator and a StrongREJECT-style rubric judge, we red-team two Google Gemini models on two benchmarks, a 30 general-harm behaviours from HarmBench and 30 financial-abuse behaviours from FinProof, each under a single-shot baseline and under IICL in four languages (English, Spanish, Hindi, Arabic). First, IICL generalizes to a second provider and is worse in finance: it lifts attack success from <=6.7% to 80-90% on HarmBench and 97-100% on FinProof, an order of magnitude above the <=24% its introducing study reported on OpenAI's GPT-5.4. Second, against the hypothesis, forcing the IICL output into a non-English language does not stack the two weaknesses, it attenuates the attack. Eleven of twelve non-English conditions score below their English baseline (sign test, p~0.003), the lone exception a ceiling tie near 100%; on the stronger model's financial set Arabic collapses from 100% to 33%. We attribute this to a relevance curse: once structure has unlocked compliance, the models produce lower-quality harmful content in lower-resource languages, which a substance-grading judge scores as partial. The pattern replicates under an independent non-Google judge (Cohen's kappa=0.86, 377 paired verdicts), and 76.6% of non-English responses were verified in-language. Jailbreak vulnerabilities are therefore not additive; the dominant residual risk is the English structural attack, most acute for financial abuse, not a multilingual one.
1 Introduction
The paper tests whether structural jailbreaks and multilingual safety weaknesses compound. It finds that IICL transfers across providers, but constraining its output to non-English languages reduces attack success.
- Introduction: IICL embeds a harmful request as the missing final cell of a short JSON reconstruction task, shifting the model toward pattern completion rather than content-level safety judgment.The mechanism is distinct from encoding, role-play, and persuasion attacks.
- Introduction: The multilingual safety gap reflects English-dominated safety training and includes both greater harmful compliance and reduced coherence in lower-resource languages.The latter is termed the relevance curse.
- Introduction: The expected compounding of structural and multilingual weaknesses is rejected: non-English output constraints consistently reduce IICL attack success rather than increasing it.The study attributes this pattern to lower-quality harmful content in lower-resource languages.
2 Background and related work
Prior work established structural in-context jailbreaks and multilingual safety gaps separately, while leaving their interaction unresolved. This paper also distinguishes its fixed IICL evaluation from adaptive attack-search methods and excludes agentic safety from study.
- In-context jailbreaks: IICL is a few-shot structural descendant of many-shot jailbreaking, using approximately 10 shots and two control tokens rather than hundreds of examples.Its original evaluation covered only OpenAI models and English, leaving cross-provider and multilingual generalization open.
- Multilingual safety: Multilingual studies show that under-represented-language prompts can bypass English-centric filters while producing both harmfulness and relevance effects, but did not combine them with structural attacks.Those studies used translation or sandwich framings.
- Automated and adaptive red-teaming: Adaptive red-teaming uses attacker models or online search, whereas this study isolates one deterministic IICL operator to avoid confounding language effects with search.The broader harness is described as a two-level bandit, but the present experiment does not use search.
- Agentic safety: AgentHarm is identified as an adjacent agentic-safety frontier, but tool-using agents are not evaluated here.The paper therefore focuses on chat-endpoint jailbreaks rather than agent behavior.
3 Threat model and hypotheses
The study evaluates IICL under a constrained black-box threat model and tests transfer, potency, and multilingual compounding. Its hypotheses treat cross-provider transfer and baseline improvement as confirmatory, while multilingual compounding is the novel test.
- Threat model: The attacker has chat-endpoint access, no model internals, and one structural prompt per probe, without multi-turn escalation or an attacker model.Because this is the weakest realistic structural attacker, observed success is a lower bound on capability.
- Hypotheses: H1 tests transfer to another provider, H2 tests whether IICL exceeds a bare single-shot baseline, and H3 tests whether non-English output raises success above English IICL.H1 and H2 are confirmatory; H3 is the novel hypothesis that the data refutes.
- Evaluation design: Table 1 reports attack success rates with Wilson 95% intervals for 30 probes per cell under a single-seed design.The caption identifies English IICL as the peak column except for the specified ceiling tie.
4 Method
The experiment compares two Google Gemini models across general-harm and financial-abuse benchmarks using single-shot and deterministic IICL prompts in four languages. Responses are graded with a StrongREJECT-style judge, while exclusions and uncertainty are reported explicitly.
- Targets and languages: The study varies English, Spanish, Hindi, and Arabic while holding the English IICL scaffold fixed and changing only the reconstructed answer cell’s target language.The models are gemini-2.5-flash and gemini-2.5-flash-lite.
- Attacks, probes, and benchmarks: HarmBench and FinProof each contribute 30 probes, with FinProof covering six financial-abuse families and the bare English behaviour serving as the single-shot floor.The deterministic IICL prompt uses a five-row benign-shot JSON scaffold with the harmful behaviour as the missing cell.
- Grading: A StrongREJECT-style rubric counts a bypass only when a response is non-refusing, on-topic, and specific/actionable, using a 0.5 threshold.Non-English responses are judged on substance regardless of language.
- Statistics and honesty: Attack success rate is the fraction of measured probes bypassed, while transport errors are excluded rather than counted as refusals.Seven of approximately 600 gradings were excluded, leaving approximately 98.8% measured; Wilson intervals and an exact sign test are reported.
5 Results
IICL strongly outperforms the single-shot baseline on both benchmarks and transfers to Google models, but its success declines when output is constrained to non-English languages. The attack is especially potent on FinProof, while the cross-provider comparison is indicative rather than controlled.
- H2 — Potency: 80.0% and 89.7% on HarmBench and 96.7% and 100% on FinProof, versus a 0–6.7% single-shot floor.The intervals are non-overlapping throughout, and FinProof is the more exposed benchmark.
- H3 — Compounding: On gemini-2.5-flash FinProof, Arabic falls from 100% to 33%, while flash-lite Arabic ties rather than exceeds its English baseline.The HarmBench decline is gentler, although its direction is unanimous.
- H1 — Generalization: IICL reaches 80–100% on Google models, compared with ≤24% on GPT-5.4 in the introducing study.The comparison is indicative because the studies used different judges and query subsets.
- H3 — Compounding: Eleven of twelve non-English IICL cells fall below their English baselines (exact sign test, p ≈0.003).The lone exception is flash-lite on FinProof, where Arabic ties English at the 96.7–100% ceiling.
6 Why the attack weakens
Non-English output weakens IICL because lower-resource responses are less complete and relevant, while language-directive failures can trigger English deflections. Thus, once IICL has resolved refusal, multilingual constraints reduce rather than compound attack success.
- Mechanism: The relevance curse lowers harmful-content quality in lower-resource languages, so a substance-grading rubric can score fluent but vague completions as non-bypasses.IICL already resolves the refusal question by construction, leaving generation quality as the limiting factor.
- Mechanism: On the financial set, ignoring the language directive and answering in English occurred 78–89% of the time for Spanish and Arabic on the stronger model.The multilingual framing can instead act as an added refusal trigger.
- Implication: Both degraded harmful content and English deflection push non-English ASR downward, despite arising through distinct routes.This explains the robust anti-compounding direction without requiring a single mechanism.
- Implication: The observed direction contradicts the harmfulness-curse prediction that lower-resource languages would increase IICL success.The paper instead points to the relevance curse as the dominant explanation for the decline.
7 Limitations and threats to validity
The study is a pilot whose claims are limited by small, single-seed cells, one new provider, and unresolved human-validation questions. Independent judging supports the pattern, but does not remove these scope boundaries.
- Scale and ceiling: With n = 30 per cell and a single seed, the pooled compounding claim rests on a twelve-cell sign test rather than any one cell.English FinProof IICL is near 100%, compressing the flash-lite comparison and widening the flash separation.
- Provider breadth: Both targets are Google models, so H1 establishes transfer to one new provider rather than universality.GPT and Claude, the direct anchors to the introducing study, are absent.
- Judge independence: An independent non-Google judge agreed with the primary judge at Cohen’s κ = 0.86 across 377 paired verdicts, but human validation remains open.The anti-compounding pattern, including the Arabic collapse, replicated under independent judging.
- Output language: The language verification covered 76.6% of 222 non-English responses, leaving output-language validation incomplete.
8 Ethics and responsible disclosure
The work is framed as defensive evaluation using public attacks and refusal-expected, low-hazard probes. It releases aggregate results and the harness rather than harmful completions.
- Ethics and responsible disclosure: The probes use standard HarmBench and low-hazard FinProof elicitation behaviours that assistants are expected to refuse.
- Ethics and responsible disclosure: The paper releases aggregate success rates and the harness, not harmful completions.
- Ethics and responsible disclosure: The authors characterize the finding as reducing attacker value while identifying the structural surface defenders should address.
9 Conclusion
IICL is potent and portable across providers, but adding a lower-resource output language blunts rather than compounds the structural attack. The findings support testing interactions directly and prioritizing the English structural exposure identified here.
- 9 Conclusion: Non-English language constraints do not compound IICL with the multilingual safety gap; they reduce attack success.The paper characterizes the residual exposure as the English structural surface rather than a multilingual combination.
- 9 Conclusion: The practical lesson is to measure interactions instead of assuming that independent jailbreak weaknesses add together.This conclusion addresses the defensive coverage question motivating the study.
- 9 Conclusion: A non-Google judge corroborates the finding with κ = 0.86, while human grading and genuinely low-resource languages remain future scale-up needs.The conclusion also notes in-language verification, GPT and Claude target anchors, and the pilot’s remaining validation needs.