PaperlayerView exact revision on arXiv

Source-linked AI summary

Arabic Safety Alignment as Selective Refusal: An Empirical Study of SFT, DPO, and Guard Calibration

Mohamad Zbib, Ammar Mohanna

arXiv:2608.29378v1cs.CLcs.AI

TL;DR

The paper evaluates Arabic safety alignment as a trade-off between benign refusal B and harmful-prompt refusal H across writing forms and models. It finds model-specific SFT trade-offs, partial Arabizi transfer, and supports deployment-specific operating-point selection.

  • Problem

    Arabic safety methods must refuse harmful prompts while remaining usable for benign and sensitive prompts across MSA, dialects, Arabizi, and noisy text.

  • Method

    The study measures benign refusal B and harmful-prompt refusal H on AraSafe and evaluates SFT, DPO, and operating points across Arabic-capable models and scripts.

  • Results

    Refusal-only SFT collapses, while selected mixed-SFT configurations reach H ≈90–93% at B = 14–23%; selected SFT raises Arabizi H for every model, but none reaches 90%.

  • Takeaways & Limitations

    Deployment should set its own safety target and retain only interventions that improve the selected operating point for the relevant model and writing form.

  • Takeaways & Limitations

    Cross-script transfer remains limited evidence because harmful supervision uses translated MSA and native-Arabizi ablations are unavailable.

Abstract

from arXiv · show

Arabic large language models must refuse harmful prompts without over-refusing benign or sensitive prompts, yet a single refusal rate hides this trade-off. We evaluate it using benign refusal B and harmful-prompt refusal H, where H measures refusal rather than harmful compliance. Across five Arabic-capable models and 130 runs on the full human-written AraSafe set, refusal-only supervised fine-tuning (SFT) collapses toward blanket refusal, whereas selected mixed-SFT configurations reach H = 90% to 93% at B = 14% to 23%; four selected configurations exceed the H = 90% target in all three runs, while Fanar does so in two of three. Direct Preference Optimization (DPO) and inference guards change B and H differently across models rather than acting as uniform upgrades. In a blinded 300-response audit, annotator binary-refusal agreement is 89.0% (kappa = 0.78); Qwen3Guard and Aya Expanse 32B reach 88.7% and 91.0% accuracy, respectively, with no conclusive paired difference. Selected SFT raises H on Arabizi for all five models, but none reaches 90%, showing only partial transfer from Modern Standard Arabic. Overall, the results support model-specific operating-point selection: set a deployment target and retain only interventions that improve it.

1 Introduction

Arabic safety alignment must balance refusing harmful prompts with remaining usable for benign and sensitive requests across MSA, dialects, Arabizi, and noisy text. The study therefore evaluates interventions on both benign refusal B and harmful-prompt refusal H, finding model-specific SFT trade-offs, heterogeneous post-training effects, and incomplete Arabizi transfer.

  • Motivation: Arabic safety spans MSA, dialects, Arabizi, and noisy text, while benign sensitive prompts can resemble harmful requests.A single refusal rate cannot distinguish selective safety from blanket refusal.
  • Evaluation framing: B measures benign refusal and H measures harmful-prompt refusal, so interventions are assessed jointly on the (B, H) plane.The study treats H as refusal rather than automatically as harmful compliance.
  • Results: H ≈90–93% at B = 14–23% is achieved by selected mixed-SFT configurations, whereas refusal-only SFT collapses toward blanket refusal.These results are presented as model-specific trade-offs rather than a universal recipe.
  • Results: DPO and inference guards move B and H differently across models rather than providing a uniform post-training upgrade.The paper reports ordering as a null result.
  • Results: Selected SFT raises Arabizi H for every model, but none reaches 90%, indicating only partial transfer across writing forms.The reported cross-script result limits conclusions drawn from MSA measurements alone.

2 Measuring and Validating Selective Refusal

The study defines operating points using separate benign and harmful-prompt refusal rates, evaluates the full held-out human-written AraSafe set, and validates automatic judgments with blinded human annotation and a cross-family judge. It searches model-specific intervention points while accounting for sampling uncertainty, seed variation, and the translated-MSA composition of training data.

  • Metrics and operating points: An operating point is a measured (B, H) pair, and the tested frontier contains non-dominated observed points.Using the illustrative H ≥90% constraint, the study selects the lowest-B feasible candidate and reports sensitivity to 85%, 90%, and 95%.
  • Evaluation set and judge: 12,077 held-out AraSafe prompts comprise 10,823 benign and 1,254 harmful examples evaluated with Qwen3Guard-Gen-4B binary-refusal labels.The set is human-written and MSA-heavy, so Sections 3–4 operating points are treated as MSA measurements.
  • Validation: 89.0% binary-refusal agreement (κ = 0.78) was observed in a blinded audit of 300 responses with two annotators and adjudication.Qwen3Guard and Aya Expanse 32B achieved 88.7% and 91.0% accuracy, respectively, against adjudicated labels.
  • Training and preference data: 10K benign and 10K harmful preference pairs support DPO, while harmful SFT and preference prompts come from translated MSA BeaverTails data.Benign SFT examples come separately from Hala-4.6M-SFT, making cross-script transfer an empirical question.
  • Uncertainty and seeds: Wilson intervals quantify prompt-sampling uncertainty conditional on automatic labels, while selected mixed-SFT configurations were repeated with seeds 43 and 44.Across models, repeated-run standard deviations span 0.13–0.57 points for B and 0.41–1.10 for H.
  • Scope of the starting points: The five models differ substantially in measured base operating points, so candidate operating points are searched separately for each model.These measurements are not treated as discovered clusters, causal diagnoses, or intervention prescriptions.

3 SFT Candidate Search

Refusal-only SFT drives models toward blanket refusal, while mixed-SFT searches reveal model-specific operating points. Selected configurations provide higher harmful-prompt refusal with substantially lower benign refusal, but results vary by model, checkpoint, and training setup.

  • Refusal-only SFT drives B toward near-total refusal after H saturates quickly.ALLaM, Fanar, and Llama cross B = 80% by step 10; both Qwen models do so by step 30.
  • Mixed-SFT ratios expose model-specific trade-offs rather than universal optima.ALLaM improves from 28.1/94.7 at 70/30 to 17.7/92.4 at 95/5, whereas Fanar reaches 8.6/79.0 at 95/5 and needs more refusal pressure.
  • Ordering does not produce a reliable benefit across model-ratio rows.Random ordering has the largest reported H − B margin in 4 of 15 rows, close to the 25% chance rate among four strategies.
  • Checkpoint choice can reduce benign refusal while preserving high harmful-prompt refusal for some models.On the 70/30 mix, AL-LaM’s B falls from 39.9 at checkpoint 50 to 19.5 at checkpoint 200 while H remains above 92; Llama over-refuses at checkpoint 50 with 35.1/96.7.
  • Selected points reach 90–93% harmful-prompt refusal at 14.0–22.6% benign refusal.These are standardized downstream comparison points, not global optima.
  • Four selected configurations exceed H = 90% in all three runs, while Fanar crosses the threshold in two of three.Fanar’s mean H = 89.95 ± 0.41 makes its rounded 90.0% seed-42 result threshold-sensitive.

4 Calibration After SFT

Post-SFT calibration compares candidate interventions on the (B, H) plane, where effects vary by model and stage rather than providing uniform improvements. Selected mixed-SFT points support downstream comparisons, while ordering evidence is treated as null and Fanar’s DPO V2 point is only a contextual proxy.

  • SFT calibration: Refusal-only SFT drives harmful-prompt refusal upward while benign refusal rises toward total refusal.Figure 2 plots B and H over training steps and marks the first crossing of B = 80%.
  • DPO: DPO moves models in opposite directions: Qwen 7B shifts from 18.6/92.4 to 13.7/89.5, whereas Llama shifts from 22.0/92.8 to 31.3/96.2.The acceptable direction depends on the deployment objective rather than a uniform upgrade rule.
  • Inference guards: Inference guards are model- and stage-dependent, lowering harmful-prompt refusal after SFT for four of five models.At base, the guard moves Fanar from 28.8/74.7 to 5.4/82.2; after SFT, only Fanar gains H, from 90.0 to 93.7, while paying an eight-point B cost.
  • Ordering: Random ordering is largest in 4 of 15 model-ratio rows, near the 25% chance rate, so the ordering comparison is treated as a null result.Reshuffling did not preserve file order as optimizer order, preventing an ordering-benefit or causal claim.
  • Operating-point comparison: Figure 5 distinguishes directly measured Base-to-SFT and SFT-to-DPO V2 movements from Fanar’s reconstructed contextual proxy.The proxy is excluded from main-text comparisons and conclusions.

5 Cross-Script Transfer and Robustness

Arabic safety behavior shifts across scripts and dialects, with Arabizi showing the sharpest base-model changes and only partial recovery after selected SFT. These controlled diagnostics support cross-form evaluation but do not establish performance on naturally occurring language.

  • Evaluation design: The 730-prompt boundary set compares behavior across MSA, Egyptian, Levantine, Arabizi, and noisy Arabic.It is a controlled synthetic diagnostic rather than a representative sample of naturally occurring user language.
  • Evaluation design: Arabizi and noisy Arabic transformations each include 22/30 prompts rated natural or mostly natural, alongside understandable-but-unnatural and invalid or meaning-changing cases.The transformation audit supports controlled stress tests but limits broad claims about natural language.
  • Base behavior across forms: Arabizi produces the sharpest base-model shifts across the tested Arabic forms.Relative to MSA, harmful-prompt refusal H falls sharply for Qwen 3B, Qwen 7B, and Fanar, while benign refusal B rises for ALLaM and Llama.
  • Selected SFT transfer: Selected SFT raises Arabizi H for all five models, but none reaches 90%.Three models remain below 55%, while ALLaM and Llama retain high benign refusal.
  • Interpreting refusal: Lower H identifies fewer harmful-prompt responses labeled as refusals, not whether the remaining responses are safe or harmful.The human taxonomy distinguishes effective, leaky, redirected, confused, and harmful or enabling responses.

6 Selecting and Interpreting Operating Points

The paper treats Arabic selective refusal as model-specific operating-point selection rather than a universal intervention problem. It recommends setting deployment targets, searching candidate methods, and validating only target-improving points across relevant conditions.

  • Operating-point selection: No single intervention dominates across models; acceptable movement on the (B, H) plane depends on the deployment objective.For Qwen 7B, selected SFT improves H but raises B, while DPO V2 lowers B but falls below H = 90%.
  • Operating-point selection: Refusal directions achieve AraSafe AUCs of 0.909–0.956 across five models, supporting local rather than universal calibration.Score scales and thresholds differ by model.
  • Deployment targets: Tightening the target from 90% to 95% costs 7.8–23.4 additional B points among feasible tested models.Fanar has no feasible tested point at 95%, so the target is a deployment choice rather than a benchmark constant.
  • Protocol: The procedure measures base (B, H), specifies deployment targets and costs, searches local candidates, and validates selected points.Validation spans seeds, independent judges, and every relevant script or dialect.
  • Conclusion: Arabic selective refusal is a model-specific operating-point selection problem, and MSA performance does not establish cross-script reliability.Refusal metrics should remain distinct from harmful-response outcomes.

7 Related Work

The paper situates Arabic selective refusal within over-refusal evaluation, Pareto safety alignment, and Arabic LLM evaluation. Its conclusions are bounded by judge uncertainty, uneven search design, translated supervision, synthetic transformations, and incomplete reproducibility details.

  • Related work: The (B, H) plane makes the refusal trade-off explicit for Arabic, connecting over-refusal evaluation with competing helpfulness and harmlessness objectives.XSTest and OR-Bench document benign over-refusal, while Panacea treats helpfulness and harmlessness as competing objectives.
  • Related work: Refusal directions and conditional activation steering motivate model-conditional control rather than a fixed intervention stack.The paper treats fine-tuning, DPO, and activation-based methods as candidate moves.
  • Arabic resources: Arabic resources span language models, safety benchmarks, safeguard evaluation, and dialectal transformation tools.Arabic diacritics are identified as an important future direction for studying refusal behavior.
  • Limitations: Judge uncertainty remains material: Wilson intervals cover prompt sampling conditional on automatic labels, not judge error, and possible family bias is not excluded.Qwen3Guard accuracy is lower on Arabizi, while the family-gap interval neither establishes nor excludes bias.
  • Limitations: Only selected SFT configurations use three seeds, and Fanar meets H ≥90% in two of three runs.Primary sweeps use seed 42.
  • Limitations: Unequal ratio coverage and exposure across models mean best-observed candidates are not causal optima.This limits causal interpretation of the search results.
  • Limitations: Harmful supervision is translated MSA, so native dialectal and Arabizi ablations are needed before attributing transfer failures to training data.The transformed prompts are synthetic, and several implementation and artifact details are unavailable.
  • Scope: The study distinguishes refusal from response safety and uses synthetic transformations only as controlled robustness diagnostics.Harmful prompts and enabling continuations are reported as outcome labels, with lower naturalness and judge agreement disclosed for Arabizi.

A.1 Preference and SFT Data

The study constructs separate translated harmful and Arabic benign data for SFT and DPO, varies SFT mixture ratios, and calibrates refusal directions on held-out splits. Several coverage, ordering, and reproducibility constraints limit interpretation of these experiments.

  • Preference data: DPO uses 10K harmful and 10K benign preference pairs, with refusals preferred over harmful completions.Benign pairs either rank two helpful answers in V1 or a helpful answer over a refusal in V2.
  • SFT data: SFT combines translated BeaverTails harmful prompts with separately sourced benign examples across ratios from 70/30 to 95/5, plus a 100% refusal control.The 100% refusal condition is a collapse control.
  • Data limitations: Coverage is not balanced across models, and exact counts per tested ratio are unavailable, so ratio is confounded with exposure or training budget.The sweep does not isolate mixture ratio from the amount of training data seen.
  • Ordering: Default reshuffling prevents file-order labels from guaranteeing realized optimizer order, so ordering is retained only as metadata and a null result.The comparison does not support a causal ordering benefit.
  • Calibration: Refusal-direction extraction uses 1,000 benign and 1,000 harmful prompts, with thresholds tuned on separate calibration splits.The calibration design separates direction extraction from threshold tuning.
  • Cross-form data: The boundary set contains 730 prompts spanning harmful, sensitive benign, and clean benign cases across multiple Arabic forms.Egyptian and Levantine are translated with AraDiCE, Arabizi is generated with GPT-4o-mini, and noisy Arabic uses a custom script.
  • Comparison limits: Fanar’s DPO V2 value is a reconstructed proxy and is excluded from direct comparisons and conclusions.It combines Fanar’s measured DPO V1 point with the median V1-to-V2 shift from four other models.

B.1 Human and Cross-Judge Audit

The audit supports approximate binary-refusal measurement, with strong human agreement but non-negligible judge error. The retained evidence also distinguishes adjudicated outcomes and naturalness agreement from refusal-label agreement.

  • 89.0% annotator binary-refusal agreement (κ = 0.78) supports consistent human labeling in the blinded audit.
  • Qwen3Guard accuracy was 86.7% on Qwen-family outputs and 90.0% otherwise, with an inconclusive paired difference.The Qwen-versus-Aya comparison had McNemar p = 0.382 and a macro-F1 difference of −2.3 points, with 95% interval [−6.4, 2.0].
  • Automatic and adjudicated labels produced B/H = 16.7/82.0 and 23.3/76.7, respectively, so judge error is not negligible.
  • The adjudicated outcomes separately characterize 44 audited harmful-prompt responses without effective refusal.
  • 85.8% agreement (κ = 0.63) was observed in the human naturalness audit of 120 transformed prompts.The audit included 30 transformed prompts per Arabic form.

C.6 Boundary Set Robustness

Boundary-set evaluation measures refusal rates across MSA, Egyptian, Levantine, Arabizi, and noisy Arabic rather than repeating the canonical AraSafe base estimates. The figures and tables distinguish Arabic-form robustness from the main benchmark measurements.

  • Boundary-set B and H are reported separately for MSA, Egyptian, Levantine, Arabizi, and noisy Arabic.The boundary set contains 730 prompts.
  • The boundary-set measurements are not repeated AraSafe base estimates, so cross-form results should not be treated as the same evaluation.For example, Fanar’s MSA 5.8/93.3 is measured on the boundary distribution.
  • Figure 7 reports base-model B and H with percentages shown above the bars.
  • Figure 8 compares selected mixed-SFT B and H operating points, while noting Fanar’s threshold sensitivity across seeds.Ordering labels identify runs only, and their ordering comparison is null.

D.2 Training and Calibration Detail

Training and calibration are presented through refusal trajectories, operating-point comparisons, calibrated thresholds, and qualitative ablations. The appendix notes that harmful prompts and enabling continuations are redacted from qualitative evidence.

  • Refusal-only and mixed-SFT trajectories report benign refusal B in the top row and harmful-prompt refusal H in the bottom row.Lines connect measured checkpoints only.
  • Figure 10 compares B and H across Base, Base+Guard, selected SFT, SFT+Guard, DPO V1, and DPO V2.Fanar DPO V2 is a proxy estimate; other points are directly measured.
  • Calibrated refusal-direction thresholds induce operating points, with benign-refusal B, harmful-prompt refusal H, and direction AUC reported per model.
  • Per-model score distributions use 10,823 benign and 1,254 harmful prompts, with dashed lines marking calibrated thresholds.Score scales are model-specific.
  • Harmful prompts and enabling ablated content are redacted, so qualitative evidence is limited to refusal-preservation or harmful-content-removal labels.
Loading 2608.29378v1…