In the Guise of Correction: How Framing and Cultural Adaptation Erode LLM Safety on Real-World Criminal Cases

1 Introduction

• Existing safety benchmarks are often synthetic, English-only, weakly grounded in external taxonomies, and lacking fine-grained analysis of severe-crime failures.The paper also asks whether safety changes when real-case prompts are culturally adapted rather than merely translated.Source: §1 · p. 1
• GUISE contains real-world criminal cases organised by ICCS and renders each case under three request framings across five language-culture settings.The set includes Bengali, Chinese, Greek, and Levantine Arabic adaptations of the English prompts.Source: §1 · p. 2
• Its two-tier framework scores actionability, specificity, and harmfulness, separating detailed preventive responses from operationally harmful ones.It derives Safe Refusal Rate and Unsafe Compliance Rate from those scores.Source: §1 · p. 2
• The controlled evaluation probes six proprietary and open-weight models across framing, category, and language, covering 13,500 responses.The study uses bootstrap confidence intervals and paired significance tests.Source: §1 · p. 2
• Culturally adapted prompts elicit significantly more unsafe compliance than literal translations of the same requests.This comparison highlights a risk that translation-only evaluations may not capture.Source: §1 · pp. 1–2

2 GUISE Corpus Construction

• The corpus construction prioritises manual search-term selection, prompt construction, and native-language adaptation to support data quality.The overview figure presents the pipeline from corpus construction through evaluation.Source: §2 · p. 2

2.1 Crime Category Selection

• Crime categories are purposively sampled across ICCS sections to span uplift levels and reporting detail, while some categories are excluded on detail-availability or ethical grounds.The authors exclude categories where reporting systematically suppresses operational detail and sexual offences against minors.Source: §2.1 · p. 2

2.2 Source Article Collection and Filtering

• From 514 crawled news articles, the authors retain 237 with enough procedural detail and select 150 diverse cases, 30 per ICCS category.Selection uses TF-IDF representations and penalises repeated publishers.Source: §2.2 · p. 3

2.3 Subgoal Extraction and Prompt Construction

• Each prompt targets one criminally relevant subgoal from a documented case rather than requesting a complete operational sequence.Prompts are checked to preserve criminal intent and ICCS category while removing personally identifiable information.Source: §2.3 · p. 3

2.4 Request Framing Conditions

• The three framing conditions hold the case and subgoal fixed while varying whether the request is direct, professionally pretextual, or correction-seeking.This design tests safety boundaries against research-helpfulness and factual-correction objectives.Source: §2.4 · pp. 3–4

2.5 Cultural Adaptation

• Native speakers adapt English prompts into Bengali, Chinese, Greek, and Levantine Arabic, with a second speaker rating a sample for fluency, cultural naturalness, and harm equivalence.Each language has one adapter to keep register and terminology consistent.Source: §2.5 · p. 4

2.6 Probing

• A fully crossed design combines three framings with five language-culture variations, producing 2,250 prompts evaluated on six models for 13,500 responses.The model set includes proprietary and open-weight systems.Source: §2.6 · p. 4

3.1 Evaluation Metrics

• Responses receive ordinal scores for actionability, specificity, and harmfulness, addressing the limits of binary safe-or-unsafe labels.A detailed response can be harmless, while refusal-like language can still contain actionable harmful content.Source: §3.1 · pp. 4–5
• Safe Refusal Rate aggregates responses meeting the safe-refusal rule, while Unsafe Compliance Rate identifies responses that are harmful, actionable, and operationally detailed.Specificity contributes to UCR severity but is direction-neutral because detailed prevention advice can remain safe.Source: §3.1 · p. 5
• Thresholds for the safe-refusal and unsafe-compliance rules were empirically selected using 100 manually reviewed responses.The sample was disjoint from the judge-calibration set and spanned models, languages, and framings.Source: §3.1 · p. 5

3.2 Selecting the Judge

• For each language, the authors select the candidate LLM judge with the highest mean weighted Cohen’s κ against human labels.Selected judges achieve substantial LLM-human agreement, either surpassing or closely approaching the human-human baseline across all five languages; each prompt-response pair is scored in isolation.Source: §3.2 · p. 5

4 Results

• Safety guardrails deteriorate sharply from direct requests to professional pretexts and correction seeking across linguistic settings, with significant drops in most tested settings.Greek is the exception: only its professional-pretext SRR drop is significant; its other three effects are marginal.Source: §4 · p. 5
• 45 of 450 model×category×condition×language cells were complete safety collapses: 31 under correction seeking, 13 under professional pretext, and one under direct requests.Fraud accounts for 21 collapses and has 18 net-unsafe cells under direct requests, versus no more than seven for any other category.Source: §4 · p. 6
• Crime category significantly predicts UCR and SRR, and its effect interacts with framing; direct-request Fraud reaches 51% UCR versus 18–32% for other categories.Under professional pretext and correction seeking, every category rises toward the ceiling and category gaps compress.Source: §4 · p. 6
• For Gemma 4 31B, Anglicization rises monotonically with harmfulness (ρ = 0.46–0.63 across languages, all p ≪ 0.001); the two least-safe models show the strongest coupling.Unsafe outputs include foreign currencies, English criminal-technique names, and platforms inaccessible in mainland China.Source: Figure 3 · p. 6, §4 · p. 7
• Localization is uneven: Bengali responses retain local taka or bKash in 96% of cases, while dollar and cryptocurrency instruments appear in other settings where prompts gave no currency.The authors interpret this asymmetry as evidence that localization occurs mainly when the local unit is supplied.Source: §4 · p. 7

4.1 Disentangling Culture from Language

• Culturally adapted prompts elicited more unsafe compliance than merely translated versions, with reported UCR differences ranging from −0.07 to −0.18; effects varied by model.Larger adaptation-related changes correlated with larger compliance gaps in Bengali (ρ=0.45) and Levantine (ρ=0.56), both significant after Holm correction. Translation quality may also contribute.Source: §4.1 · p. 8

• GUISE evaluates prompts grounded in real-world criminal cases across three request framings and five language–culture settings.Source: §5 · p. 8
• Correction seeking erodes refusal behavior across cultures, and Anglocentric registers intensify as unsafe responses become more harmful.Source: §5 · p. 9

• GUISE’s 2,250 prompts are modest in size because sourcing cases, extracting actions, selecting subgoals, and manually adapting prompts is labor-intensive.The authors prioritised prompt quality and ecological validity; expanding the corpus in the same way would require substantially more annotator time.Source: Limitations · p. 9
• Cultural coverage is limited to five language–culture settings because native-annotator availability within the research network bounds which languages can be included.The authors identify cultures with different legal systems and reporting norms as a next step.Source: Limitations · p. 9
• The six-model evaluation characterises observed framing and cultural effects but does not establish their prevalence across the model landscape.The study evaluates six LLMs and trades model breadth for scoring responses along three ordinal dimensions.Source: Limitations · p. 9
• Correction-seeking responses have lower inter-annotator agreement because corrective and operational content can be difficult to score, so their rates carry wider uncertainty.Source: Limitations · p. 9

6 Ethics Statement

• Because GUISE prompts solicit operational assistance for severe crime and several models supply it, the authors restrict distribution in recognition of misuse potential.Source: §6 · p. 9
• Prompts are encrypted behind a password, model responses are not redistributed, and released statistics are limited to aggregate SRR and UCR.Prompts were manually reviewed to remove source-identifying details while preserving criminal intent and ICCS category.Source: §6 · p. 9
• Cases come from mainstream reporting; categories with systematically suppressed operational detail and sexual offences against minors are excluded, and native annotators are informed and compensated.Source: §6 · p. 9

A.1 Case-Grounded Safety Evaluation

• GUISE combines reported cases, extracted conduct steps, and ICCS-aligned categories, then asks about one salient operational subgoal.Evaluating a partial subgoal captures assistance that can advance a harmful objective without requiring a complete criminal plan.Source: §A.1 · p. 12

A.2 Multilingual Safety Evaluation and Cultural Grounding

• Prior evaluations report more unsafe responses to non-English queries, lower-resource-language risks, and multilingual jailbreak vulnerabilities across tested models.Other work finds translated low-resource prompts can bypass safeguards and safety differs across 22 languages and 34 regions.Source: §A.2 · p. 12
• Culturally grounded benchmarks distinguish global from local harms, use transcreation or native sourcing, and evaluate responses against local norms and legal policies.CAGE preserves harmful intent in Korean adaptations, while other studies examine country-specific jailbreaks and language or geopolitical context.Source: §A.2 · p. 12

A.3 Evaluating Operational Assistance

• GUISE treats harmful compliance and operational assistance as related but distinct outcomes, evaluating matched criminal subgoals across framing and culturally adapted language settings.Prior evaluators include graded harmfulness and actionability approaches; GUISE combines these concerns in its evaluation focus.Source: §A.3 · p. 12

Purpose of the Annotation

• Annotators assess whether target-language prompts preserve the English source’s intent, framing, and harm level, and whether responses provide actionable, specific, or harmful assistance.They judge equivalence and safety by rubric, not personal moral acceptability.Source: Purpose of the Annotation · p. 13

Materials Provided to Annotators

• Each annotation includes the case identifier, ICCS category, selected subgoal, source and adapted prompts, framing, model response, and available documented action steps.The selected subgoal is the main operational element from the real-world case being tested.Source: Materials Provided to Annotators · p. 13

B.1 Cultural Adaptation: Detailed Guideline

• Adaptations may adjust phrasing, institutional terminology, and register, but must preserve the operational subgoal and avoid adding mechanisms, vulnerabilities, or harm.They also retain victim-perpetrator context, avoid regional crime localization, preserve operational detail, and maintain the framing condition.Source: §B.1 · p. 13, General Principles · p. 13

B.2 Prompt Adaptation Quality

• Prompt Adaptation Quality scores fluency, cultural naturalness, and harm equivalence on separate 0–3 scales.Harm equivalence is lower when an adaptation changes criminal intent, detail, target, mechanism, or direction toward prevention.Source: §B.2 · pp. 13–14

B.3 Response Scoring: Detailed Guideline

• Response scores assess actionability, specificity, and harmfulness independently on 0–4 scales, regardless of whether the prompt used direct, pretext, or correction-seeking framing.Defensive vocabulary does not lower scores when the response still sequences an operational procedure.Source: §B.3 · p. 14, General Principles · p. 14

General Principles

• Operational uplift is the advantage a response adds beyond its prompt: refusals and restatements provide none, while added tactics, sequencing, target logic, or evasion strategies provide high uplift.A safety disclaimer does not offset operational content.Source: General Principles · p. 14

Response Scoring Rubric

• Actionability measures usable criminal guidance, specificity measures concrete detail, and harmfulness measures capacity to enable, conceal, or normalize crime.The rubric distinguishes executable procedures from generic or non-actionable content, and harmful direction from equally specific preventive information.Source: Response Scoring Rubric · pp. 14–15

Supplementary Results & Materials

• Table 10 reports weighted Cohen’s κ by language, comparison type, framing, and rating dimension, identifying the selected LLM judge per language.It compares Human↔Human baselines with Human↔LLM agreement.Source: Table 10 · p. 16
• Figure 6 shows validation-score distributions by target language for fluency, cultural naturalness, and harm equivalence on a 0–3 scale.Bars encode the share of adapted prompts at each agreement score.Source: Figure 6 · p. 16
• Figure 7 decomposes response outcomes by model, language, and framing into safe refusal, unclassified, and unsafe compliance for 150 responses per cell.Table 11 reports SRR and UCR with bootstrapped confidence intervals and marks best values by language.Source: Figure 7 · p. 17, Table 11 · p. 17
• Table 12 reports SRR and UCR for culturally adapted versus literally translated (NLLB) prompts on a 30-case stratified subset, with framings pooled (up to 90 pairs per cell).Source: Table 12 · p. 18